Overview

Teaching: 20 min
Exercises: 0 min

Questions

• What is Data Partition

Objectives

• Learn how to split data using sklearn

Data partition: training and testing

• In Machine Learning, it is mandatory to have training and testing set. Some time a verification set is also recommended. Here are some functions for spliting training/testing set in sklearn:

• train_test_split: create series of test/training partitions
• Kfold splits the data into k groups
• StratifiedKFold splits the data into k groups based on a grouping factor.
• RepeatKfold
• ShuffleSplit
• LeaveOneOut
• LeavePOut

Due to time constraint, we only focus on train_test_split, KFolds and StratifiedKFold

3.1 Scikit-Learn data

The sklearn.datasets package embeds some small toy datasets

For each dataset, there are 4 varibles:
- **data**: numpy array of predictors/X
- **target**: numpy array of predictant/target/y
- **feature_names**: names of all predictors in X
- **target_names**: names of all predictand in y

For example:

from sklearn.datasets import load_iris
data = load_iris()
print(data.data)
print(data.target)
print(data.feature_names)
print(data.target_names)

In this example we gonna use the renowned iris flower data

from sklearn.datasets import load_iris
iris = load_iris()
X = iris.data
y = iris.target

3.2 Data spliting using train_test_split: Single fold

Here we use train_test_split to randomly split 60% data for training and the rest for testing:

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,y,train_size=0.6,random_state=123)
#random_state: int, similar to R set_seed function

3.3 Data spliting using K-fold

• This is the Cross-validation approach.
• This is a resampling process used to evaluate ML model on limited data sample.
• The general procedure:
  • Shuffle data randomly
  • Split the data into k groups For each group:
    • Split into training & testing set
    • Fit a model on each group’s training & testing set
    • Retain the evaluation score and summarize the skill of model

from sklearn.model_selection import KFold
kf10 = KFold(n_splits=10,shuffle=True,random_state=20)
for train_index, test_index in kf10.split(iris.target):
    X_train = X[train_index]
    y_train = y[train_index]
    X_test = X[test_index]
    y_test = y[test_index]
    
    model.fit(X_train, y_train) #Training the model, not running now
    y_pred = model.predict(X_test)
    print(f"Accuracy for the fold no. {i} on the test set: {accuracy_score(y_test, y_pred)}")

3.4 Data spliting using Stratified K-fold

• StratifiedKFold takes the cross validation one step further: it ensures that the target has balance class distribution.
• Look at the sample below: The target has imbalanced class distribution with 12 values of 1 and 4 values of 0. KFold will not take that into consideration when splitting the Fold

Here is the reuslt if using K-Fold:

Here is the result of using Stratified K-Fold:

from sklearn.model_selection import StratifiedKFold
kf = StratifiedKFold(n_splits=10, shuffle=True, random_state=123)
i = 1

for train_index, test_index in kf.split(iris.target):
    X_train = X[train_index]
    y_train = y[train_index]
    X_test = X[test_index]
    y_test = y[test_index]
    
    model.fit(X_train, y_train) #Training the model
    y_pred = model.predict(X_test)
    print(f"Accuracy for the fold no. {i} on the test set: {accuracy_score(y_test, y_pred)}")    

Key Points

• sklearn, data partition