Introduction to Machine Learning

Overview

Teaching: 10 min
Exercises: 0 min

Questions

What is Machine Learning

Objectives

Learng basic about Machine Learning

Supervised Learning model that we are going to learn:

Key Points

Basic Machine Learning

Introduction to Scikit Learn

Overview

Teaching: 40 min
Exercises: 0 min

Questions

What is Scikit Learn

Objectives

Master Scikit Learn for Machine Learning

2.1 What is Scikit-Learn

Scikit-learn is probably the most useful library for machine learning in Python.
The sklearn library contains a lot of efficient tools for machine learning and statistical modeling including classification, regression, clustering and dimensionality reduction.
The sklearn package contains tools for:

  - data splitting
  - pre-processing
  - feature selection
  - model tuning using resampling
  - variable importance estimation
  as well as other functionality.
  

2.2 Install `sklearn`

We have installed kernel ML_SKLN which contains the scikit-learn package in Palmetto. However for new conda environment installation, here is the command: $ pip3 install -U scikit-learn

2.3 Pre-processing using `sklearn`

There are several steps that we will use sklearn for. For preprocessing raw data, we gonna use sklearn in these tasks:

Preprocessing with missing value
Preprocessing: transform data

2.3.1 Pre-processing with missing value

Most of the time the input data has missing values (NA, NaN, Inf) due to data collection issue (power, sensor, personel).
There are three main problems that missing data causes: missing data can introduce a substantial amount of bias, make the handling and analysis of the data more arduous, and create reductions in efficiency
These missing values need to be treated/cleaned before we can use because “Garbage in => Garbage out”.
There are several ways to treat the missing values:

Method 1: remove all missing NA values

import pandas as pd
data_df = pd.DataFrame(pd.read_csv('https://raw.githubusercontent.com/vuminhtue/Machine-Learning-Python/master/data/r_airquality.csv'))
data_df.head()
data1 = data_df.dropna()

Method 2: Set NA to mean value

data2 = data_df.copy()
data2.fillna(data2.mean(), inplace=True)

import numpy as np
from sklearn.impute import SimpleImputer
imputer = SimpleImputer(missing_values=np.nan, strategy='mean')
data3 = pd.DataFrame(imputer.fit_transform(data_df))
data3.columns = data_df.columns

Note: SimpleImputer converts missing values to mean, median, most_frequent and constant.

Method 3: Use Impute to handle missing values In statistics, imputation is the process of replacing missing data with substituted values. Because missing data can create problems for analyzing data, imputation is seen as a way to avoid pitfalls involved with listwise deletion of cases that have missing values. That is to say, when one or more values are missing for a case, most statistical packages default to discarding any case that has a missing value, which may introduce bias or affect the representativeness of the results. Imputation preserves all cases by replacing missing data with an estimated value based on other available information. Once all missing values have been imputed, the data set can then be analysed using standard techniques for complete data. There have been many theories embraced by scientists to account for missing data but the majority of them introduce bias. A few of the well known attempts to deal with missing data include: hot deck and cold deck imputation; listwise and pairwise deletion; mean imputation; non-negative matrix factorization; regression imputation; last observation carried forward; stochastic imputation; and multiple imputation.

knnImpute can also be used to fill in missing value

from sklearn.impute import KNNImputer
imputer = KNNImputer(n_neighbors=2, weights="uniform")
data_knnimpute = pd.DataFrame(imputer.fit_transform(data_df))

Note:

In addition to KNNImputer, there are IterativeImputer (Multivariate imputer that estimates each feature from all the others) and MissingIndicator(Binary indicators for missing values)
More information on sklearn.impute can be found here

2.3.2 Pre-processing with Transforming data

2.3.2.1 Using Standardization

Standardization comes into picture when features of input data set have large differences between their ranges, or simply when they are measured in different measurement units for example: rainfall (0-1000mm), temperature (-10 to 40oC), humidity (0-100%), etc.
Standardition Convert all independent variables into the same scale (mean=0, std=1)
These differences in the ranges of initial features causes trouble to many machine learning models. For example, for the models that are based on distance computation, if one of the features has a broad range of values, the distance will be governed by this particular feature.

The example below use data from above:

from sklearn.preprocessing import scale
data_std = pd.DataFrame(scale(data3,axis=0, with_mean=True, with_std=True, copy=True))
# axis used to compute the means and standard deviations along. If 0, independently standardize each feature, otherwise (if 1) standardize each sample.

2.3.2.2 Using scaling with predefine range

Transform features by scaling each feature to a given range. This estimator scales and translates each feature individually such that it is in the given range on the training set, e.g. between zero and one. Formulation for this is:

X_std = (X - X.min(axis=0)) / (X.max(axis=0) - X.min(axis=0))
X_scaled = X_std * (max - min) + min

from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
#By default, it scales for (0, 1) range
data_scaler = pd.DataFrame(scaler.fit_transform(data3))

2.3.2.3 Using Box-Cox Transformation

A Box Cox transformation is a transformation of a non-normal dependent variables into a normal shape.
Normality is an important assumption for many statistical techniques; if your data isn’t normal, applying a Box-Cox means that you are able to run a broader number of tests.
The Box Cox transformation is named after statisticians George Box and Sir David Roxbee Cox who collaborated on a 1964 paper and developed the technique.

BoxCox can only be applied to stricly positive values

from sklearn.preprocessing import power_transform
data_BxCx = pd.DataFrame(power_transform(data3,method="box-cox"))
data_BxCx.columns = data3.columns

2.3.2.4 Using Yeo Johnson Transformation

While BoxCox only works with positive value, a more recent transformation method Yeo Johnson can transform both positive and negative values

data_yeo_johnson = sklearn.preprocessing.power_transform(data3,method="yeo-johnson")

import matplotlib.pyplot as plt
ax1 = plt.subplot(1,2,1)
ax1.hist(data3["Ozone"])
ax1.set_title("Original probability")
ax1.set_xlabel('Ozone')
ax1.set_ylabel('Count')
ax2 = plt.subplot(1,2,2)
ax2.hist(data_BxCx["Ozone"])
ax2.set_title("Box-Cox Transformation")
ax2.set_xlabel('Ozone')
ax2.set_ylabel('Count')
plt.show()

Key Points

sklearn

Data Partition with Scikit-Learn

Overview

Teaching: 20 min
Exercises: 0 min

Questions

What is Data Partition

Objectives

Learn how to split data using sklearn

Data partition: training and testing

In Machine Learning, it is mandatory to have training and testing set. Some time a verification set is also recommended. Here are some functions for spliting training/testing set in sklearn:
train_test_split: create series of test/training partitions
Kfold splits the data into k groups
StratifiedKFold splits the data into k groups based on a grouping factor.
RepeatKfold
ShuffleSplit
LeaveOneOut
LeavePOut

Due to time constraint, we only focus on train_test_split, KFolds and StratifiedKFold

3.1 Scikit-Learn data

The sklearn.datasets package embeds some small toy datasets

For each dataset, there are 4 varibles:
- **data**: numpy array of predictors/X
- **target**: numpy array of predictant/target/y
- **feature_names**: names of all predictors in X
- **target_names**: names of all predictand in y

For example:

from sklearn.datasets import load_iris
data = load_iris()
print(data.data)
print(data.target)
print(data.feature_names)
print(data.target_names)

In this example we gonna use the renowned iris flower data

from sklearn.datasets import load_iris
iris = load_iris()
X = iris.data
y = iris.target

3.2 Data spliting using `train_test_split`: Single fold

Here we use train_test_split to randomly split 60% data for training and the rest for testing:

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,y,train_size=0.6,random_state=123)
#random_state: int, similar to R set_seed function

3.3 Data spliting using `K-fold`

This is the Cross-validation approach.
This is a resampling process used to evaluate ML model on limited data sample.
The general procedure:
- Shuffle data randomly
- Split the data into k groups For each group:
  - Split into training & testing set
  - Fit a model on each group’s training & testing set
  - Retain the evaluation score and summarize the skill of model

from sklearn.model_selection import KFold
kf10 = KFold(n_splits=10,shuffle=True,random_state=20)
for train_index, test_index in kf10.split(iris.target):
    X_train = X[train_index]
    y_train = y[train_index]
    X_test = X[test_index]
    y_test = y[test_index]
    
    model.fit(X_train, y_train) #Training the model, not running now
    y_pred = model.predict(X_test)
    print(f"Accuracy for the fold no. {i} on the test set: {accuracy_score(y_test, y_pred)}")

3.4 Data spliting using `Stratified K-fold`

StratifiedKFold takes the cross validation one step further: it ensures that the target has balance class distribution.
Look at the sample below: The target has imbalanced class distribution with 12 values of 1 and 4 values of 0. KFold will not take that into consideration when splitting the Fold

Here is the reuslt if using K-Fold:

Here is the result of using Stratified K-Fold:

from sklearn.model_selection import StratifiedKFold
kf = StratifiedKFold(n_splits=10, shuffle=True, random_state=123)
i = 1

for train_index, test_index in kf.split(iris.target):
    X_train = X[train_index]
    y_train = y[train_index]
    X_test = X[test_index]
    y_test = y[test_index]
    
    model.fit(X_train, y_train) #Training the model
    y_pred = model.predict(X_test)
    print(f"Accuracy for the fold no. {i} on the test set: {accuracy_score(y_test, y_pred)}")    

Key Points

sklearn, data partition

Evaluation Metrics with Scikit-Learn

Overview

Teaching: 20 min
Exercises: 0 min

Questions

How do we measure the accuracy of ML model

Objectives

Learn different metrics with sklearn

4 Evaluation Metrics

Evaluation Metric is an essential part in any Machine Learning project.
It measures how good or bad is your Machine Learning model
Different Evaluation Metrics are used for Regression model (Continuous output) or Classification model (Categorical output).

4.1 Regression model Evaluation Metrics

4.1.1 Correlation Coefficient (R) or Coefficient of Determination (R2):

from sklearn import metrics
metrics.r2_score(y_test,y_pred)

4.1.2 Root Mean Square Error (RMSE) or Mean Square Error (MSE)

from sklearn import metrics
metrics.mean_squared_error(y_test,y_pred,squared=False) # RMSE
metrics.mean_squared_error(y_test,y_pred,squared=True) # MSE

4.2. Classification model Evaluation Metrics

4.2.1 Confusion Matrix

A confusion matrix is a technique for summarizing the performance of a classification algorithm.
You can learn more about Confusion Matrix here

For binary output (classification problem with only 2 output type, also most popular):

4.2.2 Accuracy

The most common metric for classification is accuracy, which is the fraction of samples predicted correctly as shown below:

from sklearn import metrics
metrics.accuracy_score(y_test,y_pred)

4.2.3 Precision

Precision is the fraction of predicted positives events that are actually positive as shown below:

4.2.4 Recall

Recall (also known as sensitivity) is the fraction of positives events that you predicted correctly as shown below:

4.2.5 F1 score

The f1 score is the harmonic mean of recall and precision, with a higher score as a better model. The f1 score is calculated using the following formula:

More information on Precision, Recall and F1 score can be found here

metrics.precision_recall_fscore_support(y_test,y_pred,average='binary')

4.2.6 AUC-ROC curve

ROC: Receiver Operating Characteristics: probability curve
AUC: Area Under The Curve: represents the degree or measure of separability.

AUC = 1: perfect prediction
AUC = 0.8: model has 80% chance to predict the right class
AUC = 0.5: worst case, model has NO accuracy in prediction (random)
AUC = 0: the model is actually reciprocating the classes

ROC Interpretation

Code to calculate FPR, TPR:

from sklearn.metrics import roc_curve
fpr, tpr, thresholds = roc_curve(y_test,y_pred)

Code to calculate AUC score:

from sklearn.metrics import roc_auc_score
auc_score = roc_auc_score(y_test,y_pred)

We will go into detail how to plot AUC-ROC curve in the next chapter with a classification problem

Key Points

sklearn, metrics

Training Machine Learning model using Regression Method

Overview

Teaching: 20 min
Exercises: 0 min

Questions

How to train a Machine Learning model using Regression method

Objectives

Learn to use different Regression algorithm for Machine Learning training

5 Supervised Learning training with Regression

5.1 For continuous output

5.1.1 Train model using Linear Regression with 1 predictor

Let use the airquality data in previous episodes:

import pandas as pd
import numpy as np
from sklearn.impute import KNNImputer
from sklearn.model_selection import train_test_split
from sklearn import metrics

data_df = pd.DataFrame(pd.read_csv('https://raw.githubusercontent.com/vuminhtue/Machine-Learning-Python/master/data/r_airquality.csv'))

imputer = KNNImputer(n_neighbors=2, weights="uniform")
data_knnimpute = pd.DataFrame(imputer.fit_transform(data_df))
data_knnimpute.columns = data_df.columns

X_train, X_test, y_train, y_test = train_test_split(data_knnimpute['Temp'],
                                                    data_knnimpute['Ozone'],
                                                    train_size=0.6,random_state=123)

Fit a Linear model using method=lm

from sklearn.linear_model import LinearRegression
model_linreg = LinearRegression().fit(X_train[:,None],y_train)

Apply trained model to testing data set and evaluate output using R-squared:

y_pred = model_linreg.predict(X_test[:,None])
metrics.r2_score(y_test,y_pred) # R^2
metrics.mean_squared_error(y_test,y_pred,squared=False) #RMSE

5.1.2 Train model using Multi-Linear Regression (with 2 or more predictors)

From the above model, the R2=0.39:

The reason is that we only build the model with 1 input Temp. In this section, we will build the model with more input Solar Radiation, Wind, Temperature:

X_train, X_test, y_train, y_test = train_test_split(data_knnimpute[['Temp','Wind','Solar.R']],
                                                    data_knnimpute['Ozone'],
                                                    train_size=0.6,random_state=123)
model_linreg = LinearRegression().fit(X_train,y_train)
y_pred2 = model_linreg.predict(X_test)

metrics.r2_score(y_test,y_pred2)
metrics.mean_squared_error(y_test,y_pred2,squared=False)

Output is therefore better with smaller RMSE and higher Rsquared at 0.5

5.1.3 Train model using Polynomial Regression

From Multi-Linear Regression, the best R2=0.5 using 3 predictors. We can slightly improve this by using Polynomial Regression

In this study, let use polynomial regression with degree of freedom=2

from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2)
X_train_poly = poly.fit_transform(data_knnimpute[['Temp','Wind','Solar.R']])
X_train, X_test, y_train, y_test = train_test_split(X_train_poly,
                                                    data_knnimpute['Ozone'],
                                                    train_size=0.6,random_state=123)
model_linreg_poly = LinearRegression().fit(X_train,y_train)
y_pred_poly = model_linreg_poly.predict(X_test)
print(metrics.r2_score(y_test,y_pred_poly))
print(metrics.mean_squared_error(y_test,y_pred_poly,squared=False))

The R2=0.58 shows improvement using polynomial regression!

5.2 For categorical output

5.2.1 Train model using Logistic Regression

Logistic regression is another technique borrowed by machine learning from the field of statistics. It is the go-to method for binary classification problems (problems with two class values).
Typical binary classification: True/False, Yes/No, Pass/Fail, Spam/No Spam, Male/Female
Unlike linear regression, the prediction for the output is transformed using a non-linear function called the logistic function.
The standard logistic function has formulation:

In this example, we create a sample data set and use logistic regression to solve it. The example is taken from here

Load library and create sample data set:

from sklearn.datasets import make_classification

# generate sample data
X, y = make_classification(n_samples=1000, n_classes=2, random_state=1)

Partitioning Data to train/test:

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=2)

Train model using Logistic Regression

from sklearn.linear_model import LogisticRegression
model_LogReg = LogisticRegression().fit(X_train, y_train)
y_pred = model_LogReg.predict(X_test)

from sklearn.linear_model import LogisticRegression
model_LogReg = LogisticRegression().fit(X_train, y_train)
# predict output:
y_pred = model_LogReg.predict(X_test)
# predict probabilities
lr_probs = model_LogReg.predict_proba(X_test)

Evaluate output with accurary level:

from sklearn import metrics
metrics.accuracy_score(y_test,y_pred)

We retrieve the accuracy = 0.834

Now compute AUC-ROC and plot curve

from sklearn.metrics import roc_curve, roc_auc_score
import matplotlib.pyplot as plt
import numpy as np

# generate a no skill prediction (majority class)
ns_probs = np.zeros(len(y_test))

# calculate scores
ns_auc = roc_auc_score(y_test, ns_probs)
lr_auc = roc_auc_score(y_test, lr_probs[:,1])
# summarize scores
print('No Skill: ROC AUC=%.3f' % (ns_auc))
print('Logistic: ROC AUC=%.3f' % (lr_auc))
# calculate roc curves
ns_fpr, ns_tpr, _ = roc_curve(y_test, ns_probs)
lr_fpr, lr_tpr, _ = roc_curve(y_test, lr_probs[:,1])
# plot the roc curve for the model
plt.plot(ns_fpr, ns_tpr, linestyle='--', label='No Skill')
plt.plot(lr_fpr, lr_tpr, marker='.', label='Logistic')
# axis labels
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
# show the legend
plt.legend()
# show the plot
plt.show()

An alternative way to plot AUC-ROC curve, using additional toolbox “scikit-plot”

pip install scikit-plot

The shorter code for using this library:

import scikitplot as skplt
skplt.metrics.plot_roc(y_test, lr_probs)
plt.show()

Key Points

Regression training

Training Machine Learning model using Tree-based model

Overview

Teaching: 20 min
Exercises: 0 min

Questions

How to train a Machine Learning model using Tree-based model

Objectives

Learn to use different Tree-based algorithm for Machine Learning training

Supervised Learning training

Train model using Decision Tree

Tree based learning algorithms are considered to be one of the best and mostly used supervised learning methods.
Tree based methods empower predictive models with high accuracy, stability and ease of interpretation
Non-parametric and non-linear relationships
Types: Categorical and Continuous

Spliting algorithm

Gini Impurity: (Categorical)
Chi-Square index (Categorical)
Cross-Entropy & Information gain (Categorical)
Reduction Variance (Continuous)

More information on how to apply the spliting algorithm to split the data can be found here

Pros & Cons

Implementation

Here we will use iris data

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

iris = load_iris()
X = iris.data
y = iris.target
X_train, X_test, y_train, y_test = train_test_split(X,y,train_size=0.6,random_state=123)

Next we will train using DecisionTree with gini splitting algorithm:

from sklearn.tree import DecisionTreeClassifier
model_DT = DecisionTreeClassifier(max_depth=3,criterion="gini").fit(X_train,y_train)

Once done, we can visualize the tree:

from sklearn import tree
tree.plot_tree(model_DT)

However, in order to have a nicer plot:

import graphviz
dot_data = tree.export_graphviz(model_DT, out_file=None,                      
                      filled=True, rounded=True,
                      feature_names=iris.feature_names,
                      special_characters=True)  
graph = graphviz.Source(dot_data) 
graph

Apply decision tree model to predic output of testing data

from sklearn import metrics
y_pred_DT = model_DT.predict(X_test)
metrics.accuracy_score(y_test,y_pred_DT)

The accuracy=0.95

More information on Decision Tree can be found here

Train model using Random Forest

Random Forest is considered to be a panacea of all data science problems. On a funny note, when you can’t think of any algorithm (irrespective of situation), use random forest!
Opposite to Decision Tree, Random Forest use bootstrapping technique to grow multiple tree
Random Forest is a versatile machine learning method capable of performing both regression and classification tasks.
It is a type of ensemble learning method, where a group of weak models combine to form a powerful model.
The end output of the model is like a black box and hence should be used judiciously.
Detail explaination
If there are M input variables, a number m<M is specified such that at each node, m variables are selected at random out of the M. The best split on these m is used to split the node. The value of m is held constant while we grow the forest.
Each tree is grown to the largest extent possible and there is no pruning.
Predict new data by aggregating the predictions of the ntree trees (i.e., majority votes for classification, average for regression).

Pros & Cons of Random Forest

Implementation of Random Forest

from sklearn.ensemble import RandomForestClassifier
model_RF = RandomForestClassifier(n_estimators=20,criterion="gini").fit(X_train,y_train)
y_pred_RF = model_RF.predict(X_test)
metrics.accuracy_score(y_test,y_pred_RF)

The accuracy=0.97

In this example, we use n_estimators=20 to grow n number of trees in the forest. We can see that Random Forest result has better prediction than Decision Tree.

More information on Random Forest can be found here

Key Points

Decision Tree, Random Forest

Training Machine Learning model using Ensemble approach

Overview

Teaching: 20 min
Exercises: 0 min

Questions

How to overcome limitation of single ML model?

Objectives

Learn to use different Ensemble ML algorithm for Machine Learning training

7.1 Why Ensemble:

Ensemble is a method in Machine Learning that combine decision from several ML models to obtain optimum output. This espisode get information from here

Source: Patheos.com

Ensemble approaches can reduce variance & Avoid Overfitting by combining results of multiple classifiers on different sub-samples

Figure. Bias & Variance Tradeoff

Bias: the difference between the model prediction & observation. High bias: model did not train well.
Variance: the variability of model prediction from one point to another. High variance: model performs really well in training but having high error rate in testing set

7.2 Train model using Ensemble Approach

Ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone. Unlike a statistical ensemble in statistical mechanics, which is usually infinite, a machine learning ensemble consists of only a concrete finite set of alternative models, but typically allows for much more flexible structure to exist among those alternatives. Here we will be learning several ensemble models:

Random Forest
Bagging
Boosting

7.3 Train model using Bagging (Bootstrap Aggregation)

The bootstrap method is a resampling technique used to estimate statistics on a population by sampling a dataset with replacement.
Bootstrap randomly create a small subsets of data from entire dataset
The subset data has similar characteristic as the entire dataset.

7.3.1 Detail explaination of Bagging

There are 3 steps in Bagging

Step 1: Here you replace the original data with new sub-sample data using bootstrapping.

Step 2: Train each sub-sample data using ML algorithm

Step 3: Lastly, you use an average value to combine the predictions of all the classifiers, depending on the problem. Generally, these combined values are more robust than a single model.

7.3.2 Implementation of Bagging

Here we use iris data set:

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

iris = load_iris()
X = iris.data
y = iris.target
X_train, X_test, y_train, y_test = train_test_split(X,y,train_size=0.6, random_state = 123)

First apply Bagging with DecisionTree model, Bagging’s parameter can be found here:

from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import BaggingClassifier
model_DT = DecisionTreeClassifier()

model_bag_DT = BaggingClassifier(base_estimator=model_DT, n_estimators=100,
                            bootstrap=True, n_jobs=-1,
                            random_state=123)
model_bag_DT.fit(X_train, y_train)

model_bag_DT.score(X_train,y_train),model_bag_DT.score(X_test,y_test)

The output accuracy from Bagging with DecisionTree for train/testing have : (1.0, 0.9666666666666667)

7.4 Train model using Boosting

http://uc-r.github.io/public/images/analytics/gbm/boosted_stumps.gif

Boosting is an approach to convert weak predictors to get stronger predictors.
Boosting follows a sequential order: output of base learner will be input to another
If a base classifier is misclassifier (red box), its weight is increased and the next base learner will classify more correctly.
Finally combine the classifier to predict result

More information on Boosting can be found here

7.4.1 Adaptive Boosting: Adaboost

Adaptive: weaker learners are tweaked by misclassify from previous classifier
AdaBoost is best used to boost the performance of decision trees on binary classification problems.
Better for classification rather than regression.
Sensitive to noise

Implementation of Adaboost

from sklearn.ensemble import AdaBoostClassifier
model_AD = AdaBoostClassifier(n_estimators=100, learning_rate=0.03).fit(X_train, y_train)

model_AD.score(X_train,y_train),model_AD.score(X_test,y_test)

The output accuracy from AdaBoost for train/testing have : (0.9333333333333333, 0.8333333333333334)

7.4.2 Gradient Boosting Machines:

Extremely popular ML algorithm
Widely used in Kaggle competition
Ensemble of shallow and weak successive tree, with each tree learning and improving on the previous

from sklearn.ensemble import GradientBoostingClassifier
model_GBM = GradientBoostingClassifier(n_estimators=100).fit(X_train,y_train)

model_GBM.score(X_train,y_train),model_GBM.score(X_test,y_test)

The output accuracy from GradientBoosting for train/testing have : (1.0, 0.9333333333333333)

7.5 Compare Bagging and Boosting technique:

7.6 Conclusions

Ensemble overcome the limitation of using only single model
Between bagging and boosting, there is no better approach without trial & error.

Key Points

Bagging, Boosting

Training Machine Learning model using Model based Prediction

Overview

Teaching: 20 min
Exercises: 0 min

Questions

What is model based prediction algorithm in ML?

Objectives

Learn to use different Model based prediction for Machine Learning training

8.1 Naive Bayes

Assuming data follow a probabilistic model
Assuming all predictors are independent (Naïve assumption)
Use Bayes’s theorem to identify optimal classifiers
More information on Aplication of Bayes’s Theorem in ML can be found here

8.1.1 Implementation Naive Bayes

Split data

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

iris = load_iris()
X = iris.data
y = iris.target
X_train, X_test, y_train, y_test = train_test_split(X,y,train_size=0.6, random_state = 123)

Train data using Naive Bayes

from sklearn.naive_bayes import GaussianNB
model_NB = GaussianNB().fit(X_train,y_train)
model_NB.score(X_train,y_train)
model_NB.score(X_test,y_test)

In addition to GaussianNB, sklearn also includes: MultinomialNB, ComplementNB, BernoulliNB, CategoricalNB. More information on Naive Bayes using sklearn can be found here

8.2 Linear Discriminent Analysis

LDA is a supervised learning model that is similar to logistic regression in that the outcome variable is categorical and can therefore be used for classification.
LDA is useful with two or more class of objects

8.2.1 Implementation LDA

Using the same iris data set, the LDA model is built:

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
model_LDA = LinearDiscriminantAnalysis().fit(X_train,y_train)
model_LDA.score(X_train,y_train)
model_LDA.score(X_test,y_test)

8.2.2 Ensemble approach (Bagging) with LDA

from sklearn.ensemble import BaggingClassifier
model_LDAbag = BaggingClassifier(base_estimator = model_LDA,n_estimators=100,
                                 bootstrap=True, n_jobs=-1,
                                 random_state=123)
model_LDAbag.fit(X_train,y_train)
model_LDAbag.score(X_train,y_train)
model_LDAbag.score(X_test,y_test)

Key Points

Naive Bayes, Linear Discriminent Analyst

Regularization and Variable Selection

Overview

Teaching: 20 min
Exercises: 0 min

Questions

Why do we need Regularization and Variable Selection in ML model

Objectives

Learn how to apply Regularization and Variable selection in ML model

One of the major aspects of training your machine learning model is to avoid overfitting (Using more parameter to best fit the training but on the other hand, failed to evaluate the testing).
The concept of balancing bias and variance, is helpful in understanding the phenomenon of overfitting

9 Regularization

In order to reduce the Model Complexity or to avoid Multi-Collinearity, one needs to reduce the number of covariates (or set the coefficient to be zero).
If the coefficients are too large, let’s penalize them to enforce them to be smaller
Regularization is a form of multilinear regression, that constrains/regularizes or shrinks the coefficient estimates towards zero.
In other words, this technique discourages learning a more complex or flexible model, so as to avoid the risk of overfitting
A simple Multi-Linear Regression look like this:

=> in which: β represents the coefficient estimates for different variables or predictors(x)

The residual sum of squares RSS is the loss function of the fitting procedure. And we need to determine the optimal coefficients 𝛽 to minimize the loss function

This procedure will adjust the β based on the training data. If there is any noise in training data, the model will not perform well for testing data. Thus, Regularization comes in and regularizes/shrinkage these 𝛽 towards zero.

There are 3 main types of Regularization.

Ridge Regression
LASSO
Elastics Nets

9.1 Ridge Regression

𝜆: Regularization Penalty, to be selected that the model minimized the error

The Ridge Regression loss function contains 2 elements: (1) RSS is actually the Ordinary Least Square (OLS) function for MLR and (2) The regularization term with 𝜆:

Selecting good 𝜆 is essential. In this case, Cross Validation method should be used
Ridge Regression enforces β to be lower but not 0. By doing so, it will not get rid of irrelevant features but rather minimize their impact on the trained model.
In statistics the coefficient esimated produced by this method is know as L2 norm
It is good practice to normalize predictors to the same scale before performing Ridge Regression (Because in OLS, the coefficients are scale equivalent)

9.1.1 Implementation

Setting up training/testing model using the Stanford’s prostate cancer data

import pandas as pd
import numpy as np
data=pd.read_csv("https://raw.githubusercontent.com/vuminhtue/Machine-Learning-Python/master/data/prostate_data.csv")
ind_train = data["train"]=="T"
data = data.drop(["train"],axis=1)
X_train = data.drop(["lpsa"],axis=1)[ind_train]
X_test = data.drop(["lpsa"],axis=1)[~ind_train]
y_train = data["lpsa"][ind_train]
y_test = data["lpsa"][~ind_train]

Predict using Ridge Regression method and Cross Validation approach:

from sklearn.linear_model import RidgeCV
from sklearn.metrics import mean_squared_error as mse

n_lambda = 100
lambdas = np.logspace(-2,6, n_lambda)

MSE_train = []
MSE_test = []
coefs = []

for ld in lambdas:
    ridgecv = RidgeCV(alphas = [ld], normalize = True)
    model_RR = ridgecv.fit(X_train, y_train)
    y_predRR_cv_train = model_RR.predict(X_train)
    y_predRR_cv_test  = model_RR.predict(X_test)    
    MSE_train.append(mse(y_train,y_predRR_cv_train))
    MSE_test.append(mse(y_test,y_predRR_cv_test))    
    coefs.append(model_RR.coef_)

coef_df = pd.DataFrame(coefs)
coef_df.columns = X_train.columns

Plotting the Mean Square Error for Training and Testing dataset based on 𝜆 variation

import matplotlib.pyplot as plt

fig, ax = plt.subplots(1, 2, figsize=(16, 8), constrained_layout=False)

ax1 = plt.subplot(221)
ax1.scatter(np.log10(lambdas), MSE_train,color="red")
ax1.set_title("Training Set")
ax2 = plt.subplot(222)
ax2.scatter(np.log10(lambdas), MSE_test,color="red")
ax2.set_title("Testing Set")

ax1.set_xlabel("log($\\lambda$)")
ax2.set_xlabel("log($\\lambda$)")
ax1.set_ylabel('MSE')
ax2.set_ylabel('MSE')

plt.show()

Plotting the coefficient of different predictors based on 𝜆

ax = plt.gca()
for i in range(0,coef_df.columns.size):
    ax.plot(np.log10(lambdas), coef_df.iloc[:,i])
    
ax.legend(coef_df.columns)
#ax.set_xscale('log')
plt.xlabel("log($\\lambda$)")
plt.ylabel('Coefficients')
plt.title('Ridge Regression Coefficients')
plt.axis('tight')
plt.show()

The plot shows different coefficients for all predictors with 𝜆 variation.

Ridge Regression’s pros: the pros of RR method over OLS is rooted in the bias variance trade-off. As when 𝜆 increases, the flexibility of RR fit decreases, hence decrease the variance but increase the bias
Ridge Regression’s cons: β never be 0, so all predictors are included in the final model. Therefore, it is not good for best feature selection.

9.2 LASSO: Least Absolute Shrinkage & Selection Operator

In order to overcome the cons issue in Ridge Regression, the LASSO is introduced with the similar shrinkage parameter, but the different is not in square term of the coefficient but only absolute value
Similar to Ridge Regression, LASSO also shrink the coefficient, but force coefficients to be equal to 0. Making it ability to perform feature selection
In statistics the coefficient esimated produced by this method is know as L1 norm

9.2.1 Implementation

Predict using Lasso method:

from sklearn.linear_model import Lasso

n_lambda = 100
lambdas1 = np.logspace(-6,0, n_lambda)

MSE_train = []
MSE_test = []
coefs = []
for ld in lambdas1:
    lassocv = Lasso(alpha=ld)
    model_LS = lassocv.fit(X_train, y_train)
    y_predLS_cv_train = model_LS.predict(X_train)
    y_predLS_cv_test = model_LS.predict(X_test)
    MSE_train.append(mse(y_train,y_predLS_cv_train))
    MSE_test.append(mse(y_test,y_predLS_cv_test))
    coefs.append(model_LS.coef_)    

Plotting the Mean Square Error for Training and Testing dataset based on 𝜆 variation

fig, ax = plt.subplots(1, 2, figsize=(16, 8), constrained_layout=False)


ax1 = plt.subplot(221)
ax1.scatter(np.log10(lambdas1), MSE_train,color="red")
ax1.set_title("Training Set")
ax2 = plt.subplot(222)
ax2.scatter(np.log10(lambdas1), MSE_test,color="red")
ax2.set_title("Testing Set")

ax1.set_xlabel("log($\\lambda$)")
ax2.set_xlabel("log($\\lambda$)")
ax1.set_ylabel('MSE')
ax2.set_ylabel('MSE')

plt.show()

Plotting the coefficient of different predictors based on 𝜆

coef_df = pd.DataFrame(coefs)
coef_df.columns = X_train.columns

ax = plt.gca()
for i in range(0,coef_df.columns.size):
    ax.plot(np.log10(lambdas1), coef_df.iloc[:,i])
    
ax.legend(coef_df.columns,bbox_to_anchor = (1.05, 0.6))
#ax.set_xscale('log')
plt.xlabel("log($\\lambda$)")
plt.ylabel('Coefficients')
plt.title('LASSO Coefficients')
plt.axis('tight')
plt.show()

The plot shows different coefficients for all predictors with 𝜆 variation. Depending on 𝜆 values that the β varying and it can be 0 at certain point.

9.3 Elastic Nets

Elastic Nets Regularization is a method that includes both LASSO and Ridge Regression. Its formulation for the loss function is as following:

𝛼=0: pure Ridge Regression
𝛼=1: pure LASSO
0<𝛼<1: Elastic Nets

9.3.1 Implementation

from sklearn.linear_model import ElasticNet

MSE_train = []
MSE_test = []
coefs = []
for ld in lambdas1:
    Elastic_cv = ElasticNet(alpha=ld,l1_ratio=0.5)
    model_EN = Elastic_cv.fit(X_train, y_train)
    y_predEN_cv_train = model_EN.predict(X_train)
    y_predEN_cv_test = model_EN.predict(X_test)
    MSE_train.append(mse(y_train,y_predEN_cv_train))
    MSE_test.append(mse(y_test,y_predEN_cv_test))
    coefs.append(model_EN.coef_)

The ElasticNet mixing parameter, with 0 <= l1_ratio <= 1.
For l1_ratio = 0 the penalty is an L2 penalty (Ridge Regression).
For l1_ratio = 1 it is an L1 penalty (LASSO).
For 0 < l1_ratio < 1, the penalty is a combination of L1 and L2.

Key Points

Regularization, Ridge Regression, LASSO, Elastic Nets

Dimension Reduction

Overview

Teaching: 20 min
Exercises: 0 min

Questions

What happen when there are lots of covariates?

Objectives

Learn how to apply PCA in ML model

10 Principal Component Analysis

Handy with large data
Where many variables correlate with one another, they will all contribute strongly to the same principal component
Each principal component sums up a certain percentage of the total variation in the dataset
More Principal Components, more summarization of the original data sets

10.1 PCA formulation

For example, we have 3 data sets: X, Y, Z
We need to compute the covariance matrix M for the 3 data set:

in which, the covariance value between 2 data sets can be computed as:

For the Covariance matrix M, we will find m eigenvectors and m eigenvalues

- Given mxm matrix, we can find m eigenvectors and m eigenvalues
- Eigenvectors can only be found for square matrix.
- Not every square matrix has eigenvectors
- A square matrix A and its transpose have the same eigenvalues but different eigenvectors
- The eigenvalues of a diagonal or triangular matrix are its diagonal elements.
- Eigenvectors of a matrix A with distinct eigenvalues are linearly independent.

Eigenvector with the largest eigenvalue forms the first principal component of the data set … and so on …*

10.2 Implementation

Here we gonna use iris data set:

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

import numpy as np
import pandas as pd
iris = load_iris()
X = iris.data
y = pd.DataFrame(iris.target)
y['Species']=pd.Categorical.from_codes(iris.target, iris.target_names)
X_train, X_test, y_train, y_test = train_test_split(X,y,train_size=0.6,random_state=123)

X_train_scaled = StandardScaler().fit_transform(X_train)
X_test_scaled = StandardScaler().fit_transform(X_test)

10.2.1 Compute PCA using sklearn:

from sklearn.decomposition import PCA
pca = PCA(n_components=4)
PCs = pca.fit_transform(X_train_scaled)
PCs = pd.DataFrame(PCs,columns = ['PC1','PC2','PC3','PC4'])

We can see that PCs computed from sklearn package are similar to newpca computed from using eigen vectors

10.2.2 Explained Variance

The explained variance tells you how much information (variance) can be attributed to each of the principal components.

pca.explained_variance_ratio_

In this example: the PC1(0.74) and PC2 (0.21) consume 0.95 percent of explained variance. Therefore, using 2 Principal Components should be good enough

10.2.3 Application of PCA model in Machine Learning:

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score as acc_score

pca = PCA(n_components=2) #We choose number of principal components to be 2

X_train_pca = pca.fit_transform(X_train_scaled)
X_test_pca = pd.DataFrame(pca.transform(X_test_scaled))
X_test_pca.columns=['PC1','PC2']

print(pca.explained_variance_ratio_)

# Use random forest to train model
model_RF = RandomForestClassifier(n_estimators=20,criterion="gini",random_state=1234).fit(X_train_pca, y_train['Species'])
y_pred_RF = model_RF.predict(X_test_pca)
acc_score(y_test['Species'],y_pred_RF)

Plotting the testing result with indicator of Wrong prediction

import matplotlib.pyplot as plt

ax = plt.gca()

targets = np.unique(y_pred_RF)
colors = ['r', 'g', 'b']

for target, color in zip(targets,colors):
    indp = y_pred_RF == target
    ax.scatter(X_test_pca.loc[indp, 'PC1'], X_test_pca.loc[indp, 'PC2'],c = color)

# Ploting the Wrong Prediction
ind = y_pred_RF!=np.array(y_test['Species'])
ax.scatter(X_test_pca.loc[ind, 'PC1'],X_test_pca.loc[ind, 'PC2'],c = 'black')

#axis control
ax.legend(['setosa','versicolor','virginica','Wrong Prediction'])  
ax.set_title("Testing set from Random Forest using PCA 2 components")
ax.set_xlabel('PC1')
ax.set_ylabel('PC2')

plt.show()

Key Points

PCA

Neural Network

Overview

Teaching: 20 min
Exercises: 0 min

Questions

How to use Neural Network in Machine Learning model

Objectives

Learn how to use ANN in ML model

11. Neural Network

Neural network is a series of algorithms that endeavors to recognize underlying relationships in a set of data through a process that mimics the way the human brain operates.
Neuron is a basic unit in a nervous system and is the most important component of the brain.
In each Neuron, there is a cell body (node), dendrite (input signal) and axon (output signal to other neuron).
If a Neuron received enough signal, it is then activated to decide whether or not it should transmitt the signal to other neuron or not.

Formulation of Neural Network

Here, x1,x2....xn are input variables. w1,w2....wn are weights of respective inputs.
b is the bias, which is summed with the weighted inputs to form the net inputs. 
Bias and weights are both adjustable parameters of the neuron.
Parameters are adjusted using some learning rules. 
The output of a neuron can range from -inf to +inf.
The neuron doesn’t know the boundary. So we need a mapping mechanism between the input and output of the neuron. 
This mechanism of mapping inputs to output is known as Activation Function.

Activation functions:

import matplotlib.pyplot as plt
import numpy as np

xrange = np.linspace(-2, 2, 200)

plt.figure(figsize=(7,6))

plt.plot(xrange, np.maximum(xrange, 0), label = 'ReLU')
plt.plot(xrange, np.tanh(xrange), label = 'Hyperbolic Tangent')
plt.plot(xrange, 1 / (1 + np.exp(-xrange)), label = 'Sigmoid/Logistic')
plt.plot(xrange, xrange, label = 'Linear')
plt.plot(xrange, np.heaviside(xrange, 0.5), label = 'Step')
plt.legend()
plt.title('Neural network activation functions')
plt.xlabel('Input value (x)')
plt.ylabel('Activation function output')

plt.show()

Neural Network formulation: Multi-Layer Perceptron (MLP) Multi-layer Perceptron (MLP) is a supervised learning algorithm. Given a set of features X = x1, x2, ... xm, and target y, MLP can learn a non-linear function approximator for either classification or regression.

Between the input and the output layer, there can be one or more non-linear layers, called hidden layers. Figure below shows a one hidden layer MLP with scalar output.

The advantages of Multi-layer Perceptron:

Capability to learn non-linear models.
Capability to learn models in real-time (on-line learning) using partial_fit.

The disadvantages of Multi-layer Perceptron:

MLP with hidden layers have a non-convex loss function where there exists more than one local minimum. Therefore different random weight initializations can lead to different validation accuracy.
MLP requires tuning a number of hyperparameters such as the number of hidden neurons, layers, and iterations.
MLP is sensitive to feature scaling.

11.1. Type of Neural Network Multi-Layer Perceptron in sklearn

There are 2 main types of MLP in sklearn, depending on the model output:

MLPClassifier: for Classification problem
MLPRegressor: for Regression problem

11.2. Implementation with Classification problem

Here we use iris data for Classification problem

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler

import numpy as np
import pandas as pd
iris = load_iris()
X = iris.data
y = pd.DataFrame(iris.target)
y['Species']=pd.Categorical.from_codes(iris.target, iris.target_names)
X_train, X_test, y_train, y_test = train_test_split(X,y,train_size=0.6,random_state=123)

scaler = MinMaxScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

The Class MLPClassifier implements a multi-layer perceptron (MLP) algorithm that trains using Backpropagation. There are lots of parameters in MLPClassifier:

hidden_layer_sizes which is the number of hidden layers and neurons for each layer. Default=(100,) for example hidden_layer_sizes=(100,) means there is 1 hidden layers used, with 100 neurons. for example hidden_layer_sizes=(50,20) means there are 2 hidden layers used, the first layer has 50 neuron and the second has 20 neurons.
solver lbfgs, sgd, adam. Default=adam
activation identity, logistic, tanh, relu. Default=’relu`

More information can be found here

from sklearn.neural_network import MLPClassifier
model_NN = MLPClassifier(hidden_layer_sizes = (50,20),solver='lbfgs',activation='relu',random_state=123).fit(X_train_scaled, y_train['Species'])
model_NN.score(X_test_scaled,y_test['Species'])

11.3. Implementation with Regression problem

Class MLPRegressor implements a multi-layer perceptron (MLP) that trains using backpropagation with no activation function in the output layer, which can also be seen as using the identity function as activation function.
Therefore, it uses the square error as the loss function, and the output is a set of continuous values.

Here we use airquality data from Regression espisode:

import pandas as pd
import numpy as np
from sklearn.impute import KNNImputer
from sklearn.model_selection import train_test_split

data_df = pd.DataFrame(pd.read_csv('https://raw.githubusercontent.com/vuminhtue/Machine-Learning-Python/master/data/r_airquality.csv'))

imputer = KNNImputer(n_neighbors=2, weights="uniform")
data_knnimpute = pd.DataFrame(imputer.fit_transform(data_df))
data_knnimpute.columns = data_df.columns

X_train, X_test, y_train, y_test = train_test_split(data_knnimpute[['Temp','Wind','Solar.R']],
                                                    data_knnimpute['Ozone'],
                                                    train_size=0.6,random_state=123)

Fit MLPRegressor model

from sklearn.neural_network import MLPRegressor
model_NN = MLPRegressor(hidden_layer_sizes = (50,20),solver='lbfgs',activation='relu',max_iter=1000).fit(X_train,y_train)
model_NN.score(X_test,y_test)

11.4. Tips on using MLP

Multi-layer Perceptron is sensitive to feature scaling, so it is highly recommended to scale your data.
There is built-in regularization parameter alpha (Default 0.0001). To find reasonable alpha, it’s best to use GridSearchCV, usually in the range 10.0**-np.arange(1, 7)
Empirically, we observed that L-BFGS converges faster and with better solutions on small datasets. For relatively large datasets, however, Adam is very robust. It usually converges quickly and gives pretty good performance. SGD with momentum or nesterov’s momentum, on the other hand, can perform better than those two algorithms if learning rate is correctly tuned.
Since backpropagation has a high time complexity, it is advisable to start with smaller number of hidden neurons and few hidden layers for training.
The loss function for Classifier is Cross-Entropy while for Regression is Square-Error

Key Points

ANN

Support Vector Machine

Overview

Teaching: 20 min
Exercises: 0 min

Questions

How to use Support Vector Machine in Machine Learning model

Objectives

Learn how to use SVM in ML model

12. Support Vector Machine

A support vector machine is a very important and versatile machine learning algorithm,
It is capable of doing linear and nonlinear classification, regression and outlier detection.
It is preferred over other classification algorithms because it uses less computation and gives notable accuracy.
It is good because it gives reliable results even if there is less data
The objective of the support vector machine (SVM) algorithm is to find a hyperplane in an N-dimensional space that distinctly classifies the data points.

12.1. Applications of Support Vector Machine:

12.2. Explanation

To separate the two classes of data points, there are many possible hyperplanes that could be chosen

SVM’s objective is to find a plane that has the maximum margin, i.e the maximum distance between data points of both classes. Maximizing the margin distance provides some reinforcement so that future data points can be classified with more confidence.

Example of hyperplane in 2D and 3D position:

Support vectors (SVs) are data points that are closer to the hyperplane and influence the position and orientation of the hyperplane. Using SVs to maximize the margin of the classifier. Removing SVs will change the position of the hyperplane. These are the points that help us build our SVM.

12.3. SVM’s kernel

12.3.1. For linear separable data, it is quite straight forward to create a hyperplane to distinguish them

12.3.2. For linearly non-separable data, SVM makes use of kernel tricks to make it linearly separable.

The concept of transformation of non-linearly separable data into linearly separable is called Cover’s theorem - “given a set of training data that is not linearly separable, with high probability it can be transformed into a linearly separable training set by projecting it into a higher-dimensional space via some non-linear transformation”.
Kernel tricks help in projecting data points to the higher dimensional space by which they became relatively more easily separable in higher-dimensional space.
Kernel tricks also known as Generalized dot product.
Kernel tricks are the way of calculating dot product of two vectors to check how much they make an effect on each other.
According to Cover’s theorem the chances of linearly non-separable data sets becoming linearly separable increase in higher dimensions.
Kernel functions are used to get the dot products to solve SVM constrained optimization.

The following kernel trick compared different kernel ‘linear’ , ’poly’ , ‘rbf’ , ‘sigmoid’:

12.4. Implementation

Here we use the regular iris dataset with Classification problem

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler

import numpy as np
import pandas as pd
iris = load_iris()
X = iris.data
X = X[:,2:4]
y = iris.target

X_train, X_test, y_train, y_test = train_test_split(X,y,train_size=0.6,random_state=123)

Fit Support Vector Classifier model

from sklearn.svm import SVC
from mlxtend.plotting import plot_decision_regions
import matplotlib.pyplot as plt
names = ["Linear SVM", "RBF SVM", "Poly SVM", "Sigmoid SVM"]
classifiers = [
    SVC(kernel="linear"),
    SVC(kernel="rbf"),
    SVC(kernel="poly"),
    SVC(kernel="sigmoid")]

i = 1
figure = plt.figure(figsize=(27, 5))
cm = plt.cm.jet

for name, clf in zip(names, classifiers):
    ax = plt.subplot(1,4, i)
    clf.fit(X_train, y_train)
    ax = plot_decision_regions(X=X_train, 
                      y=y_train,
                      clf=clf)
    ax.set_xlabel(iris.feature_names[2], size=14)
    ax.set_ylabel(iris.feature_names[3], size=14)
    ax.set_title(name, size=20)
    handles, labels = ax.get_legend_handles_labels()
    ax.legend(handles,iris.target_names)
    i+=1
plt.show()

In this model, C is the regularization parameter Default C=1. The strength of the regularization is inversely proportional to C. Must be strictly positive.

12.5. Tips on using SVM

Setting C=1 is reasonable choice for default. If you have a lot of noisy observations you should decrease it: decreasing C corresponds to more regularization.
More information here

12.6. Pros of SVM

High stability due to dependency on support vectors and not the data points.
Does not get influenced by Outliers.
No assumptions made of the datasets.
Numeric predictions problem can be dealt with SVM.

12.7. Cons of SVM

Blackbox method.
Inclined to overfitting method.
Very rigorous computation.

Key Points

SVM

K-Nearest Neighbour

Overview

Teaching: 20 min
Exercises: 0 min

Questions

How to use K-Nearest Neighbour in Machine Learning model

Objectives

Learn how to use KNN in ML model

13. K-Nearest Neighbour

Simplicity but powerful and fast for certain task
Work for both classification and regression
Named as Instance Based Learning; Non-parametrics; Lazy learner
Work well with small number of inputs

13.1. Explanation

In KNN, the most important parameter is the K group and the distance computed between points.
Euclide distance:

13.2. Implementation

Here we gonna use the iris dataset again:

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

import numpy as np
import pandas as pd
iris = load_iris()
X = iris.data
y = iris.target

X_train, X_test, y_train, y_test = train_test_split(X,y,train_size=0.6,random_state=123)

Train the model using KNN model with 3 nearest neighbors

from sklearn.neighbors import KNeighborsClassifier
model_KNN = KNeighborsClassifier(n_neighbors=3).fit(X_train,y_train)

model_KNN.score(X_train,y_train)
model_KNN.score(X_test,y_test)

13.3. Other similar models from sklearn.neighbors:

KNeighborsRegressor: KNN estimators for Regression problem with continuous data
NearestCentroid
and more

Key Points

KNN

Unsupervised Learning

Overview

Teaching: 20 min
Exercises: 0 min

Questions

What is Unsupervised Learning in Machine Learning model

Objectives

Learn how to use K-mean clustering in ML model

14. Unsupervised Learning

No labels are given to the learning algorithm leaving it on its own to find structure in its input.
Unsupervised learning can be a goal in itself (discovering hidden patterns in data) or a means towards an end (feature learning).

Used when no feature output data
Often used for clustering data

Typical method:

K-means clustering
Hierarchical clustering
Ward clustering
Partition Around Median (PAM)

14.1. K-means clustering

14.1.1. Explanation of K-means clustering method:

Given a set of data, we choose K=2 clusters to be splited:

First select 2 random centroids (denoted as red and blue X)

Compute the distance between 2 centroid red X and blue X with all the points (for instance using Euclidean distance) and compare with each other. 2 groups are created with shorter distance to 2 centroids

Now recompute the new centroids of the 2 groups (using mean value of all points in the same groups):

Compute the distance between 2 new centroids and all the points. We have 2 new groups:

Repeat the last 2 steps until no more new centroids created. The model reach equilibrium:

14.1.2. Example with K=3

14.1.3. Implementation

Here we use the iris data set with only predictors

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

import numpy as np
import pandas as pd
iris = load_iris()
X = iris.data

Apply Kmeans and plotting

from sklearn.cluster import KMeans
import matplotlib.pyplot as plt

model_KMeans = KMeans(n_clusters=3)
model_KMeans.fit(X)

plt.scatter(X[:,2],X[:,3],c=model_KMeans.labels_)
plt.xlabel(iris.feature_names[2])
plt.ylabel(iris.feature_names[3])
plt.title('KMeans clustering with 3 clusters')
plt.show()

14.1.4. How to find optimal K values:

14.1.4.1. Elbow approach

Similar to KNN method for supervised learning, for K-means approach, we are able to use Elbow approach to find the optimal K values.
The Elbow approach ues the Within-Cluster Sum of Square (WSS) to measure the compactness of the clusters:

The optimal K-values can be found from the Elbow using method=”wss”:

wss = []
for k in range(1,10):
    model = KMeans(n_clusters=k).fit(X)
    wss.append(model.inertia_)
    
plt.scatter(range(1,10),wss)
plt.plot(range(1,10),wss)
plt.xlabel("Number of Clusters k")
plt.ylabel("Within Sum of Square")
plt.title("Optimal number of clusters based on WSS Method")
plt.show()

14.1.4.2. Gap-Statistics approach

Developed by Prof. Tibshirani et al in Stanford
Applied to any clustering method (K-means, Hierarchical)
Maximize the Gap function:

E*n: expectation under a sample size of n from the reference distribution

Installation:

This version of Gap Statistics is not official. Until the moment of writing this documentation, no official Gap Statistics has been released in Python. We use the version from milesgranger’s github

pip install git+git://github.com/milesgranger/gap_statistic.git
pip install gapstat-rs

Implement Gap-Statistics:

from gap_statistic import OptimalK

optimalK = OptimalK(n_jobs=1) # No parallel
n_clusters = optimalK(X[:,1:4], cluster_array=np.arange(1, 15))
print('Optimal clusters: ', n_clusters)

Plot Gap-Statistics:

import matplotlib.pyplot as plt
plt.plot(optimalK.gap_df.n_clusters, optimalK.gap_df.gap_value, linewidth=3)
plt.scatter(optimalK.gap_df[optimalK.gap_df.n_clusters == n_clusters].n_clusters,
            optimalK.gap_df[optimalK.gap_df.n_clusters == n_clusters].gap_value, s=250, c='r')
plt.grid(True)
plt.xlabel('Cluster Count')
plt.ylabel('Gap Value')
plt.title('Gap Values by Cluster Count')
plt.show()

14.2. Comparison between different clustering methods in sklearn:

This is example from sklearn
The source code for image below can be found here

Key Points

K-mean

Mini-Project

Overview

Teaching: 10 min
Exercises: 180 min

Questions

What do you learn from Machine Learning workshop using scikit-learn?

Objectives

Learn how to apply Machine Learning into your real project

15. MINI PROJECT

Description of Mini Project:

This Mini Project evaluates the ability of you working on a sample Data Science project from scratch.
The project is about Supervised Machine Learning only
It requires you to download data, clean data, split into training/testing and apply any machine learning model and analyse the output.
You can also use what you have learn during the Deep Learning workshop as well (Optional) like CNN, RNN or just traditional Fully connected Dense layer.

Requirement:

You can write the script in Jupyter Notebook and save it as your_username.ipynb and send it to my email: tuev@clemson.edu. You can also upload to your github page and share to me
Note: in order to get the Certificate of Attendence, you should send me the ipynb script on or before 10/31/2021.

Data

You can use any data in your field of expertise (most preferred method). If so, please send along the dataset with the ipynb file
You can also use Kaggle data which can be found here (second preferred method). Please elaborate in the ipynb how did you retrieve the data
You can use use Titanic data from our repo. Description for Titanic data can be found here (third preferred method)
You can use any of the Toy dataset from scikit learn which can be found here, except the iris data.

Method

In the ipynb solution file I would like to see the following:

Clearly state the objective of the mini-project on Supervised Machine Learning
Brief explanation about the data that you will be using: source, predictors, predictand
Type of ML model output: Continuous or Classification?
Read in the data
Clean & Standardized the input data if needed
Split data to training/testing (You can also use Cross Validation if needed, not required)
You can use any Regularization (variable selection) or PCA if needed (not required)
Construct Machine Learning model to training set and explain why do you want to use that algorithm (any model is fine for me)
Apply Machine Learning model to predict the output from testing set
Evaluate the output using any of the given method in chapter 4
Confirm if your ML model is good or bad?

Key Points

Self-project

Introduction to Machine Learning using Python

Introduction to Machine Learning

Overview

Key Points

Introduction to Scikit Learn

Overview

2.1 What is Scikit-Learn

2.2 Install sklearn

2.3 Pre-processing using sklearn

2.3.1 Pre-processing with missing value

2.3.2 Pre-processing with Transforming data

2.3.2.1 Using Standardization

2.3.2.2 Using scaling with predefine range

2.3.2.3 Using Box-Cox Transformation

2.3.2.4 Using Yeo Johnson Transformation

Key Points

Data Partition with Scikit-Learn

Overview

Data partition: training and testing

3.1 Scikit-Learn data

3.2 Data spliting using train_test_split: Single fold

3.3 Data spliting using K-fold

3.4 Data spliting using Stratified K-fold

Key Points

Evaluation Metrics with Scikit-Learn

Overview

4 Evaluation Metrics

4.1 Regression model Evaluation Metrics

4.1.1 Correlation Coefficient (R) or Coefficient of Determination (R2):

4.1.2 Root Mean Square Error (RMSE) or Mean Square Error (MSE)

4.2. Classification model Evaluation Metrics

4.2.1 Confusion Matrix

4.2.2 Accuracy

4.2.3 Precision

4.2.4 Recall

4.2.5 F1 score

4.2.6 AUC-ROC curve

Key Points

Training Machine Learning model using Regression Method

Overview

5 Supervised Learning training with Regression

5.1 For continuous output

5.1.1 Train model using Linear Regression with 1 predictor

5.1.2 Train model using Multi-Linear Regression (with 2 or more predictors)

5.1.3 Train model using Polynomial Regression

5.2 For categorical output

5.2.1 Train model using Logistic Regression

Key Points

Training Machine Learning model using Tree-based model

Overview

Supervised Learning training

Train model using Decision Tree

Spliting algorithm

Pros & Cons

Implementation

Train model using Random Forest

Detail explaination

Pros & Cons of Random Forest

Implementation of Random Forest

Key Points

Training Machine Learning model using Ensemble approach

Overview

7.1 Why Ensemble:

7.2 Train model using Ensemble Approach

7.3 Train model using Bagging (Bootstrap Aggregation)

7.3.1 Detail explaination of Bagging

7.3.2 Implementation of Bagging

7.4 Train model using Boosting

7.4.1 Adaptive Boosting: Adaboost

Implementation of Adaboost

7.4.2 Gradient Boosting Machines:

7.5 Compare Bagging and Boosting technique:

7.6 Conclusions

Key Points

Training Machine Learning model using Model based Prediction

Overview

8.1 Naive Bayes

8.1.1 Implementation Naive Bayes

8.2 Linear Discriminent Analysis

8.2.1 Implementation LDA

2.2 Install `sklearn`

2.3 Pre-processing using `sklearn`

3.2 Data spliting using `train_test_split`: Single fold

3.3 Data spliting using `K-fold`

3.4 Data spliting using `Stratified K-fold`