This lesson is being piloted (Beta version)

Machine Learning with R

Introduction to Data Science workflow with R

Overview

Teaching: 10 min
Exercises: 0 min
Questions

  • What is the overall workflow of Data Science

Objectives

  • Learn to know how to perform a Data Science project with R

Slide1

Slide2

Slide4

Slide5

Slide6

Slide7

Slide8

Slide9

Slide10

Slide11

Slide12

Slide13

Key Points

  • Data Science, R programming language, caret, supervised learning, unsupervised learning

Introduction to Caret

Overview

Teaching: 40 min
Exercises: 0 min
Questions

  • What is Caret

Objectives

  • Master Caret package for Machine Learning

2.1 What is Caret

image

The caret package (short for Classification And REgression Training) is a set of functions that attempt to streamline the process for creating predictive models. The package contains tools for:

data splitting pre-processing feature selection model tuning using resampling variable importance estimation as well as other functionality.

There are many different modeling functions in R. Some have different syntax for model training and/or prediction. The package started off as a way to provide a uniform interface the functions themselves, as well as a way to standardize common tasks (such parameter tuning and variable importance).

The current release version can be found on CRAN and the project is hosted on github. Caret was developed by Max Kuhn Here only touch some of the very basic command that is useful for our Machine Learning class.

caret cheatsheet

2.2 Why using Caret

2.3 Install caret

In R console:

install.packages("caret", dependencies = c("Depends", "Suggests"))

In R studio:

Select Tools\Install Packages and select caret from CRAN

Once installed, load the caret package to make sure that it works:

library(caret)

2.4 Pre-processing using caret

There are several steps that we will use caret for. For preprocessing raw data, we gonna use caret in these tasks:

Pre-processing with missing value

Here we use preProcess function from caret to perform bagImpute (Bootstrap Aggregation Imputation):

library(caret)
PreImputeBag <- preProcess(airquality,method="bagImpute")
DataImputeBag <- predict(PreImputeBag,airquality)

In addition to bagImpute, we also can use knnImpute (K-Nearest Neighbour Imputation) knnImpute can also be used to impute missing value, however, it standardize the data after Imputing:

MData <- airquality[,-c(1,5,6)]
PreImputeKNN <- preProcess(MData,method="knnImpute",k=5)
DataImputeKNN <- predict(PreImputeKNN,MData)

#Convert back to original scale
RescaleDataM <- t(t(DataImputeKNN)*PreImputeKNN$std+PreImputeKNN$mean)

Note bagImpute is more powerful and computational cost than knnImpute

Visualize the input data using corrplot

We can also quickly visualize the cross correlation between the input data to see the relationship:

dmatrix <- cor(DataImputeBag[,1:4])
corrplot(dmatrix)

image

Key Points

  • Caret

Data Partition with caret

Overview

Teaching: 20 min
Exercises: 0 min
Questions

  • What is Data Partition

Objectives

  • Learn how to split data using caret

3 Data partition: training and testing

image

Due to time constraint, we only focus on createDataPartition and createFolds

3.1 Data spliting using Data Partition

Here we use createDataPartition to randomly split 60% data for training and the rest for testing: image

ind1 <- createDataPartition(y=iris$Species,p=0.6,list=FALSE,times=1)
#list=FALSE, prevent returning result as a list
#times=1 to create the resample size. Default value is 1.
training <- iris[ind1,]
testing  <- iris[-ind1,]

3.2 Data spliting using K-fold: Cross validation approach

The procedure has a single parameter called k that refers to the number of groups that a given data sample is to be split into. As such, the procedure is often called k-fold cross-validation. When a specific value for k is chosen, it may be used in place of k in the reference to the model, such as k=10 becoming 10-fold cross-validation. image

fitControl <- trainControl(method="cv", number=10)
# train the model
model <- train(Species~., data=training, 
               trControl=fitControl, method="lda")
# summarize results
print(model)
predict1 <- predict(model,testing)

3.3 Other Cross-Validation approach

method: The resampling method: “boot”, “cv”, “LOOCV”, “LGOCV”, “repeatedcv”, “timeslice”, “none” and “oob”

More information on model tuning using caret can be found here

Key Points

  • Caret

Evaluation Metrics with caret

Overview

Teaching: 40 min
Exercises: 0 min
Questions

  • How do we measure the accuracy of ML model

Objectives

  • Learn different metrics with caret

4 Evaluation Metrics

4.1 Regression model Evaluation Metrics

4.1.1 Correlation Coefficient (R) or Coefficient of Determination (R2):

image

cor(prediction,testing)
cor.test(prediction,testing)

4.1.2 Root Mean Square Error (RMSE) or Mean Square Error (MSE)

image

The postResample function gives RMSE, R2 and MAE at the same time:

postResample(prediction,testing$Ozone)

4.2. Classification model Evaluation Metrics

4.2.1 Confusion Matrix

For binary output (classification problem with only 2 output type, also most popular):

image

 confusionMatrix(predict,testing)

Key Points

  • Caret

Supervised Learning with Continuous Output

Overview

Teaching: 20 min
Exercises: 0 min
Questions

  • How to train a Machine Learning model using Regression method

Objectives

  • Learn to use different Regression algorithm for Machine Learning training

5 Supervised Learning training with Continuous output

Here we use the R sampled data named airquality with some missing values.

data(airquality)

5.1 Pre-processing data and treat missing value

Check missing value

sum(is.na(airquality))

Impute missing value using Bagging approach

PreImputeBag <- preProcess(airquality,method="bagImpute")
airquality_imp <- predict(PreImputeBag,airquality)

5.2 Visualize the important data

library(GGally)
ggpairs(airquality_imp,aes(colour=factor(Month)))

image

5.3 Split data into training and testing

indT <- createDataPartition(y=airquality_imp$Ozone,p=0.6,list=FALSE)
training <- airquality_imp[indT,]
testing  <- airquality_imp[-indT,]

Let’s use all inputs data (except Month/Day) for modeling

5.4 Train model and predict with different algorithm

5.4.1 Train model using Multi Linear Regression modeling: ‘method=lm’

Here we will use 3 input variables as inputs to predict the output. We will need to standardize the input using flag preProcess=c(“center”,”scale”)

ModFit_lm <- train(Ozone~Solar.R+Wind+Temp,data=training,
                 preProcess=c("center","scale"),
                 method="lm")
predict_lm <- predict(ModFit_lm,testing)

5.4.2 Train model using Stepwise Linear Regression

Stepwise linear regression is a method of regressing multiple variables while simultaneously removing those that aren’t important.

Stepwise regression essentially does multiple regression a number of times, each time removing the weakest correlated variable. At the end you are left with the variables that explain the distribution best. The only requirements are that the data is normally distributed (or rather, that the residuals are), and that there is no correlation between the independent variables (known as collinearity).

Option is to use AIC or BIC criterion for removing the weak variable

ModFit_SLR <- train(Ozone~Solar.R+Wind+Temp,data=training,method="lmStepAIC")
predict_SLR <- predict(ModFit_SLR,testing)

5.4.3 Train model using Polynomial Regression

Polynomial regression is a form of regression analysis in which the relationship between the independent variable x and the dependent variable y is modelled as an n^th degree polynomial in x.

Polynomial regression fits a nonlinear relationship between the value of x and the corresponding conditional mean of y

image

In this study, let use polynomial regression with degree of freedom=3 with function poly

ModFit_poly <- train(Ozone~poly(Solar.R,3)+poly(Wind,3)+poly(Temp,3),data=training,
                     preProcess=c("center","scale"),
                     method="lm")
predict_poly <- predict(ModFit_poly,testing)

5.4.4 Train model using Principal Component Regression

Linear Regression using the output of a Principal Component Analysis (PCA). PCR is skillful when data has lots of highly correlated predictors

ModFit_PCR <- train(Ozone~Solar.R+Wind+Temp,data=training,method="pcr")
predict_PCR <- predict(ModFit_PCR,testing)

5.4.5 Train model using Decision Tree

Spliting algorithm

Pros & Cons

image

ModFit_rpart <- train(Ozone~Solar.R+Wind+Temp,data=training,method="rpart",
                      parms = list(split = "gini"))
predict_rpart <- predict(ModFit_rpart,testing)

Want fancier plot?

library(rattle)
fancyRpartPlot(ModFit_rpart$finalModel)

5.4.6 Train model using Random Forest

image

Detail explaination

Pros & Cons of Random Forest

image

ModFit_rf <- train(Ozone~Solar.R+Wind+Temp,data=training,method="rf",prox=TRUE)
predict_rf <- predict(ModFit_rf,testing)

5.4.7 Train model using Artificial Neural Network

image image

image

Here, x1,x2....xn are input variables. w1,w2....wn are weights of respective inputs.
b is the bias, which is summed with the weighted inputs to form the net inputs. 
Bias and weights are both adjustable parameters of the neuron.
Parameters are adjusted using some learning rules. 
The output of a neuron can range from -inf to +inf.
The neuron doesn’t know the boundary. So we need a mapping mechanism between the input and output of the neuron. 
This mechanism of mapping inputs to output is known as Activation Function.

image

image

image

library(neuralnet)
smax <- apply(training,2,max)
smin <- apply(training,2,min)
trainNN <- as.data.frame(scale(training,center=smin,scale=smax-smin))
testNN <- as.data.frame(scale(testing,center=smin,scale=smax-smin))
ModNN <- neuralnet(Ozone~Solar.R+Wind+Temp,trainNN, hidden=3,linear.output = T)
plot(ModNN)

predict_ann <- compute(ModNN,testNN)
# Rescale to original:
predict_ann_rescale <- predict_ann$net.result*(smax-smin)[1]+smin[1]

image

5.5 Evaluate model output

For continuous, we use postResample:

postResample(predict_lm,testing$Ozone)
postResample(predict_SLR,testing$Ozone)
postResample(predict_PCR,testing$Ozone)
postResample(predict_poly,testing$Ozone)
postResample(predict_rpart,testing$Ozone)
postResample(predict_rf,testing$Ozone)
postResample(predict_ann_rescale,testing$Ozone)

Key Points

  • Regression training

Supervised Learning with Categorical Output

Overview

Teaching: 20 min
Exercises: 0 min
Questions

  • How to train a Machine Learning model with categorical output

Objectives

  • Learn to use different Regression algorithm for Machine Learning training

6 Supervised Learning training with Categorical output

Here we use the R sampled data named iris

data(iris)

6.1 Pre-processing data and treat missing value

Check missing value

sum(is.na(iris))

There are no missing value so we go ahead with visualize the important data

6.2 Visualize the important data

library(GGally)
ggpairs(iris,aes(colour=Species))

image

6.3 Split data into training and testing

indT <- createDataPartition(y=iris$Species,p=0.6,list=FALSE)
training <- iris[indT,]
testing  <- iris[-indT,]

Let’s use all inputs data for modeling

6.4 Train model and predict with different algorithm

6.4.1 Train model using Linear Discriminant Analyst:

ModFit_lda <- train(Species~.,data=training,
                 preProcess=c("center","scale"),
                 method="lda")
predict_lda <- predict(ModFit_lda,testing)

6.4.2 Train model using Naive Bayes

  ModFit_nb <- train(Species~.,data=training,method="nb")
  predict_nb <- predict(ModFit_nb,testing)

6.4.3 Train model using Gradient Boosting Machine

ModFit_GBM <- train(Species~.,data=training,method="gbm",verbose=FALSE)
predict_GBM <- predict(ModFit_GBM,testing)

6.4.4 Train model using Random Forest

ModFit_rf <- train(Species~.,data=training,method="rf",prox=TRUE)
predict_rf <- predict(ModFit_rf,testing)

6.4.5 Train model using Artificial Neural Network

ModNN <- neuralnet(Species~.,training, hidden=c(4,3),linear.output = FALSE)
plot(ModNN)

predict_ann <- compute(ModNN,testing)
# Rescale to original:
yann=data.frame("yhat"=ifelse(max.col(predict_ann$net.result)==1, "setosa",
                              ifelse(max.col(predict_ann$net.result)==2, "versicolor", "virginica")))

image

6.5 Evaluate model output

For continuous, we use postResample:

confusionMatrix(predict_lda,testing$Species)
confusionMatrix(predict_nb,testing$Species)
confusionMatrix(predict_GBM,testing$Species)
confusionMatrix(predict_rf,testing$Species)
confusionMatrix(yann$yhat,testing$Species)

Key Points

  • categorical output

Unsupervised Learning

Overview

Teaching: 20 min
Exercises: 0 min
Questions

  • What is Unsupervised Learning in Machine Learning model

Objectives

  • Learn how to use K-mean clustering in ML model

7 Unsupervised Learning

image image

image

image

image

image

image

image

7.1.2 Example with K=3

image

image

7.1.3 Implementation

library(ggplot2)
library(factoextra)
library(purrr)
data(iris)
ggplot(iris,aes(x=Sepal.Length,y=Petal.Width))+
      geom_point(aes(color=Species))
set.seed(123)
km <- kmeans(iris[,3:4],3,nstart=20)

table(km$cluster,iris$Species)
fviz_cluster(km,data=iris[,3:4])

image

7.2 How to find optimal K values:

7.2.1 Elbow approach

The optimal K-values can be found from the Elbow using method=”wss”:

fviz_nbclust(iris[,3:4], kmeans, method = "wss")

image

7.2.2 Gap-Statistics approach

image

E*n: expectation under a sample size of n from the reference distribution image

image

library(cluster)
# B is number of Monte Carlo bootstrap samples
gap_stat <- clusGap(iris[,3:4], FUN = kmeans, nstart=20, K.max = 10, B = 50)
fviz_gap_stat(gap_stat)

image

Key Points

  • K-mean

Regularization and Variable Selection

Overview

Teaching: 20 min
Exercises: 0 min
Questions

  • Why do we need Regularization and Variable Selection in ML model

Objectives

  • Learn how to apply Regularization and Variable selection in ML model

image

8.1 Regularization

=> in which: β represents the coefficient estimates for different variables or predictors(x)

The residual sum of squares RSS is the loss function of the fitting procedure. And we need to determine the optimal coefficients 𝛽 to minimize the loss function

image

This procedure will adjust the β based on the training data. If there is any noise in training data, the model will not perform well for testing data. Thus, Regularization comes in and regularizes/shrinkage these 𝛽 towards zero.

There are 3 main types of Regularization.

For simplicity, I only introduce LASSO for Regularization method

LASSO: Least Absolute Shrinkage & Selection Operator

image

Setting up training/testing model:

library(caret)
#library(ElemStatLearn)=> available for R package > 3.6.2
prostate=read.csv("https://raw.githubusercontent.com/vuminhtue/Machine-Learning-Python/master/data/prostate_data.csv")


set.seed(123)
indT <- which(prostate$train==TRUE)
training <- prostate[indT,]
testing  <- prostate[-indT,]

library(PerformanceAnalytics)
chart.Correlation(training[,-10])

Splitting to training/testing

library(glmnet)
library(plotmo)
y <- training$lpsa
x <- training[,-c(9,10)]
x <- as.matrix(x)

LASSO computation

cvfit_LASSO    <- cv.glmnet(x,y,alpha=1)
plot(cvfit_LASSO)

log(cvfit_LASSO$lambda.min)
log(cvfit_LASSO$lambda.1se)

coef(cvfit_LASSO,s=cvfit_LASSO$lambda.min)
coef(cvfit_LASSO,s=cvfit_LASSO$lambda.1se)

image

Fit_LASSO <- glmnet(x,y,alpha=1)
plot_glmnet(Fit_LASSO,label=TRUE,xvar="lambda",
            col=seq(1,8),,grid.col = 'lightgray')

image The plot shows different coefficients for all predictors with 𝜆 variation. Depending on 𝜆 values that the β varying and it can be 0 at certain point.

Using 𝜆.1se, we obtain reasonable result:

> predict_LASSO <- predict(cvfit_LASSO,newx=xtest,s="lambda.1se")
> postResample(predict_LASSO,testing$lpsa)
     RMSE  Rsquared       MAE 
0.6783357 0.6096333 0.5030956

8.2 Dimension Reduction using PCA

8.2.1 Explanation

in which, the covariance value between 2 data sets can be computed as: image

- Given mxm matrix, we can find m eigenvectors and m eigenvalues
- Eigenvectors can only be found for square matrix.
- Not every square matrix has eigenvectors
- A square matrix A and its transpose have the same eigenvalues but different eigenvectors
- The eigenvalues of a diagonal or triangular matrix are its diagonal elements.
- Eigenvectors of a matrix A with distinct eigenvalues are linearly independent.

Eigenvector with the largest eigenvalue forms the first principal component of the data set … and so on …*

8.2.2 Implementation

8.2.2.1 Compute PCA using eigenvector:

library(PerformanceAnalytics)
data(mtcars)
#Ignore vs & am (PCA works good with numeric data )
datain <- mtcars[,c(1:7,10:11)]
chart.Correlation(datain)
cin <- cov(scale(datain))
ein <- eigen(cin)
newpca <-   -scale(datain) %*% ein$vectors

8.2.2.2 Compute PCA using built-in function:

mtcars.pca <- prcomp(datain,center=TRUE,scale=TRUE)
summary(mtcars.pca)

8.2.2.3 A nice way to plot PCA:

Install ggbiplot package:

library(devtools)
install_github("vqv/ggbiplot")

library(ggbiplot)
ggbiplot(mtcars.pca)
ggbiplot(mtcars.pca, labels=rownames(mtcars))
ggbiplot(mtcars.pca,ellipse=TRUE,  labels=rownames(mtcars))

mtcars.country <- c(rep("Japan", 3), rep("US",4), rep("Europe", 7),rep("US",3), "Europe", rep("Japan", 3), rep("US",4), rep("Europe", 3), "US", rep("Europe", 3))

ggbiplot(mtcars.pca,ellipse=TRUE,labels=rownames(mtcars),groups = mtcars.country)

image

8.2.2.4 Application of PCA model in Machine Learning:

data(mtcars)
set.seed(123)
datain <- mtcars[,c(1:7,10:11)]

indT <- createDataPartition(y=datain$mpg,p=0.6,list=FALSE)
training <- datain[indT,]
testing  <- datain[-indT,]

preProc <- preProcess(training[,-1],method="pca",pcaComp = 1)
trainPC <- predict(preProc,training[,-1])
testPC  <- predict(preProc,testing[,-1])

traindat<- cbind(training$mpg,trainPC)
testdat <- cbind(testing$mpg,testPC)

names(traindat) <- c("mpg","PC1")
names(testdat)  <- names(traindat) 

modFitPC<- train(mpg~.,method="lm",data=traindat)

predictand <- predict(modFitPC,testdat)
postResample(testing$mpg,as.vector(predictand))

Key Points

  • Regularization, Ridge Regression, LASSO, Elastic Nets, PCA

Kaggle online competition: Supervised Learning

Overview

Teaching: 20 min
Exercises: 0 min
Questions

  • How to participate in a Kaggle online compeition

Objectives

  • Download Kaggle data and apply some algorithm technique that you have learnt to solve the actual data

9. Kaggle online competition: Supervised Learning

This is a perfect competition for data science students who have completed an online course in machine learning and are looking to expand their skill set before trying a featured competition.

https://www.kaggle.com/c/house-prices-advanced-regression-techniques/overview

image

Project description:

Ask a home buyer to describe their dream house, and they probably won’t begin with the height of the basement ceiling or the proximity to an east-west railroad. But this playground competition’s dataset proves that much more influences price negotiations than the number of bedrooms or a white-picket fence.

With 79 explanatory variables describing (almost) every aspect of residential homes in Ames, Iowa, this competition challenges you to predict the final price of each home.

For simpilicity: I downloaded the data for you and put it here: https://github.com/vuminhtue/SMU_Data_Science_workflow_R/tree/master/data/Kaggle_house_prices

9.1 Understand the data

There are 4 files in this folder:

Objective:

9.2 Create the Rmd format with following Data Science workflow:

Step 1: Load library, Load data

Step 2: Select variables.

Step 3: Create partition for the data

Step 4: Apply 1 ML algorithm to the data and calculate prediction

Step 5: Evaluate the model output

Step 6: Knit the documentation

! Solution

Key Points

  • Kaggle

Kaggle online competition: Unsupervised Learning

Overview

Teaching: 20 min
Exercises: 0 min
Questions

  • How to participate in a Kaggle online compeition

Objectives

  • Download Kaggle data and apply some algorithm technique that you have learnt to solve the actual data

10. Kaggle online competition: Unsupervised Learning

In previous chapter, you have worked with Supervised Learning data, now in this chapter, let’s confront with another type of ML problem, which is Unsupervised Learning

https://www.kaggle.com/majyhain/height-of-male-and-female-by-country-2022

image

Project description: The metric system is used in most nations to measure height.Despite the fact that the metric system is the most widely used measurement method, we will offer average heights in both metric and imperial units for each country.To be clear, the imperial system utilises feet and inches to measure height, whereas the metric system uses metres and centimetres.Although switching between these measurement units is not difficult, countries tend to choose one over the other in order to maintain uniformity.

For simpilicity: I downloaded the data for you and put the data table here: https://raw.githubusercontent.com/vuminhtue/SMU_Data_Science_workflow_R/master/data/Heights/Height%20of%20Male%20and%20Female%20by%20Country%202022.csv

10.1 Understand the data

There is only 1 csv file: Height of Male and Female by Country 2022

The dataset contains six columns: • Rank • Country Name • Male height in Cm • Female height in Cm • Male height in Ft • Female height in Ft

Objective:

10.2 Create the Rmd format with following Data Science workflow:

Step 1: Load library, Load data

Step 2: Find the optimimum number of clusters

Step 3: Use Kmeans clustering to classify clusters

Step 4: Visualize the difference

Step 5: Knit the documentation

!> Solution

Key Points

  • Kaggle

Fundamental Text Mining using R

Overview

Teaching: 20 min
Exercises: 0 min
Questions

  • What is Text Mining and how to use R to work with that

Objectives

  • Learn some terminology in Text Mining with R

11. Text Mining Introduction

We will walk through the Text Mining using an example with Halloween scary monster poem using the most popular Text Mining package tm

Set working directory

rm(list=ls())
setwd('/users/tuev/SMU/Workshop/SMU/R_Text_Mining/')

Opening an example

library(tm)
library(ggplot2)
mytxt <- readLines('https://raw.githubusercontent.com/vuminhtue/SMU_Machine_Learning_R/master/data/monster_mash.txt')

Library tm

A framework for text mining applications within R.The tm package offers

functionality for managing text documents, abstracts the process of document manipulation and eases the usage of heterogeneous text formats in R. The package has integrated database back-end support to minimize memory demands. An advanced meta data management is implemented for collections of text documents to alleviate the usage of large and with meta data enriched document sets.

The package provides native support for reading in several classic file formats (e.g. plain text, PDFs, or XML files).

tm provides easy access to preprocessing and manipulation mechanisms such as whitespace removal, stemming, or stopword deletion. Further a generic filter architecture is available in order to filter documents for certain criteria, or perform full text search. The package supports the export from document collections to term-document matrices.

Corpus Corpora

Corpora are collections of documents containing (natural language) text. In packages which employ the infrastructure provided by package tm, such corpora are represented via the virtual S3 class Corpus: such packages then provide S3 corpus classes extending the virtual base class (such as VCorpus provided by package tm itself).

A corpus can have two types of metadata

# Convert to Corpus format
dc <- VCorpus(VectorSource(mytxt))

Text manipulation

Now that we have converted text data to Corpus format, we can reduce the text dimension using some popular builtin function in tm library

Lower case

Here we enforce that all of the text is lowercase. This makes it easier to match cases and sort words.

Notice we are assigning our modified column back to itself. This will save our modifications to our Data Frame

dc <- tm_map(dc,content_transformer(tolower))
for (i in 1:12) print(dc[[i]]$content)

Remove Punctuation

Here we remove all punctuation from the data. This allows us to focus on the words only as well as assist in matching.

dc <- tm_map(dc,removePunctuation)
for (i in 1:12) print(dc[[i]]$content)

Remove Stopwords

Stopwords are words that are commonly used and do little to aid in the understanding of the content of a text. There is no universal list of stopwords and they vary on the style, time period and media from which your text came from. Typically, people choose to remove stopwords from their data, as it adds extra clutter while the words themselves provide little to no insight as to the nature of the data. For now, we are simply going to count them to get an idea of how many there are.

dc <- tm_map(dc,removeWords,stopwords("english"))
for (i in 1:12) print(dc[[i]]$content)

Strip white space

dc <- tm_map(dc,stripWhitespace)
for (i in 1:12) print(dc[[i]]$content)

Stemming

Stemming is the process of removing suffices, like “ed” or “ing”.

dc_stemmed <- tm_map(dc,stemDocument)
for (i in 1:12) print(dc_stemmed[[i]]$content)

As we can see “eyes” became “eye”, which could help an analysis, but “castle” became “castl” which is less helpful.

Lemmatization

library(textstem)
dc_lemmatize <- tm_map(dc,content_transformer(lemmatize_strings))
for (i in 1:12) print(dc_lemmatize[[i]]$content)

Notice how we still caught “eyes” to “eye” but left “castle” as is.

TermDocumenMatrix

The text data is unstructure data and you need to convert that into structured data for ML analyses. There are two approaches to convert unstructured data to a structured form:

  1. Bag of Words Model
  2. Vector Space Model

1. Bag of Words model

In the Bag of Words model, the text document is represented by a bag of words. The model can be represented as a table containing the frequency of the words and the words themselves. For instance, consider a text document containing the following sentences-

The bag of words for the text document is: image

2. Vector Space Model

Vector Space Model (VSM) is the generalization of the Bag of Words model. In the Vector Space model, each document from the corpus is represented as a multidimensional vector. Each unique term from the corpus represents one dimension of the vector space. A term can be a single word or sequence of words (ngrams). The number of unique terms in the corpus determines the dimension of the vector space.

In VSM, the corpus is represented in the form of the Term Document Matrix. Term Document Matrix represents documents vectors in matrix form in which the rows correspond to the terms in the document, columns correspond to the documents in the corpus and cells correspond to the weights of the terms.

DTM (Document term matrix) is obtained by taking the transpose of TDM. In DTM, the rows correspond to the documents in the corpus and the columns correspond to the terms in the documents and the cells correspond to the weights of the terms.

No we can apply the TF-IDF using control for shorter script Note: We are using the original dc data:

dc_tdm <- TermDocumentMatrix(dc_lemmatize)
print(dc_tdm$dimnames$Terms)

3. Computing Terms’ Weights

There are various approaches for determining the terms’ weights. The simple and frequently used approaches include:-

3.1. Binary weights 3.2. Term Frequency (TF) 3.3. Inverse Document Frequency (IDF) 3.4. Term Frequency-Inverse Document Frequency (TF-IDF)

3.1. Binary weights

In the case of binary weights, the weights take the values- 0 or 1 where 1 reflects the presence and 0 reflects the absence of the term in a particular document. For instance,

D1: Text mining is to find useful information from text. D2: Useful information is mined from the text. D3: Dark came.

image

3.2. Term Frequency (TF)

In the case of the Term Frequency, the weights represent the frequency of the term in a specific document. The underlying assumption is that the higher the term frequency in a document, the more important it is for that document.

TF(t)= c(t,d) c(t,d)- the number of occurences of the term t in the document d.

3.3. Inverse Document Frequency (IDF)

In the case of IDF, the underlying idea is to assign higher weights to unusual terms, i.e., to terms that are not so common in the corpus. IDF is computed at the corpus level, and thus describes corpus as a whole, not individual documents. It is computed in the following way:

IDF(t)=1+log(N/df(t))

N: number of documents in the corpus Df(t): number of documents with the term t

For instance, suppose there are 100 documents in the corpus and 10 documents contain the term text.

Then, IDF(text)=1+log(100/10)=1+1=2

3.4. TF-IDF

In the case of TF-IDF, the underlying idea is to value those terms that are not so common in the corpus (relatively high IDF), but still have some reasonable level of frequency (relatively high TF). It is the most frequently used metric for computing term weights in a vector space model.

General formula for computing TF-IDF:

TF-IDF(t)=TF(t)*IDF(t)

One popular ‘instantiation’ of this formula:

TF-IDF(t)= tf(t)*log(N/df(t))

Implementing in R code

The following terms are computed using TermDocumentMatrix function

dc_tdm <- TermDocumentMatrix(dc_lemmatize)
print(dc_tdm$dimnames$Terms)

To analyze the term frequency:

CorpusMatrix <- as.matrix(dc_tdm)
sortedMatrix <- sort(rowSums(CorpusMatrix),decreasing=TRUE)
dfCorpus <- data.frame(word = names(sortedMatrix),freq=sortedMatrix)
head(dfCorpus,5)

Bar plot

w <- rowSums(CorpusMatrix)
w_sub <- subset(w,w>3)
barplot(w_sub,las=3, col=rainbow(20))

Word Clouds

library(wordcloud2)
wordcloud2(data=dfCorpus,size=1.6,shape='star')

reate N-Grams and plot histogram using RWeka

library(RWeka)
dfNgrams <- data.frame(text=sapply(dc_lemmatize,as.character),stringsAsFactors = FALSE)
uniGramToken <- NGramTokenizer(dfNgrams,Weka_control(min=1,max=1))
biGramToken <- NGramTokenizer(dfNgrams,Weka_control(min=2,max=2))
triGramToken <- NGramTokenizer(dfNgrams,Weka_control(min=3,max=3))

uniGrams <- data.frame(table(uniGramToken))
biGrams  <- data.frame(table(biGramToken))
triGrams <- data.frame(table(triGramToken))

uniGrams <- uniGrams[order(uniGrams$Freq,decreasing=TRUE),]
colnames(uniGrams) <- c('Word','Frequency')
biGrams <- biGrams[order(biGrams$Freq,decreasing=TRUE),]
colnames(biGrams) <- c('Word','Frequency')
triGrams <- triGrams[order(triGrams$Freq,decreasing=TRUE),]
colnames(triGrams) <- c('Word','Frequency')

uniGrams_s <- uniGrams[1:10,]
biGrams_s  <- biGrams[1:10,]
triGrams_s <- triGrams[1:10,]

library(ggplot2)
plotUniGrams <- ggplot(head(uniGrams,10),aes(x=Frequency,y=reorder(Word,Frequency),fill=Word))+
                  geom_bar(stat='identity')+
                  scale_fill_brewer(palette="Spectral")+
                  geom_text(aes(x=Frequency,label=Frequency,vjust=1))+
                  labs(x="Frequency (%)",y="Words",title="UniGrams Frequency")
                  
                  
plotUniGrams

plotBiGrams <- ggplot(head(biGrams,10),aes(x=Frequency,y=reorder(Word,Frequency),fill=Word))+
                  geom_bar(stat='identity')+
                  scale_fill_brewer(palette="Spectral")+
                  geom_text(aes(x=Frequency,label=Frequency,vjust=1))+
                  labs(x="Frequency (%)",y="Words",title="BiGrams Frequency")
plotBiGrams

plotTriGrams <- ggplot(head(triGrams,10),aes(x=Frequency,y=reorder(Word,Frequency),fill=Word))+
                  geom_bar(stat='identity')+
                  scale_fill_brewer(palette="Spectral")+
                  geom_text(aes(x=Frequency,label=Frequency,vjust=1))+
                  labs(x="Frequency (%)",y="Words",title="TriGrams Frequency")
plotTriGrams

Word Clouds for BiGrams and TriGrams

wordcloud2(data=uniGrams,size=1.6)
wordcloud2(data=biGrams,size=1.6)
wordcloud2(data=triGrams,size=1.6)

Exercise

DATA

For this comparison, our data set is a sample of 19 documents from the Gutenberg Collection. The Project Gutenberg is a volunteer effort to digitize and archive cultural works as well as to “encourage the creation and distribution of eBooks”, found in 1971 and is the oldest digital library.

The Gutenberg collection can be accessed in R using the “gutenbergr” package.

The data for this example is a data frame with four columns and nineteen rows. Each row represents a document. The text of the document is in the full_text variable. The remaining columns - id, author, and title - provide metadata for the given document.

The gutenbergr package provides access to the public domain works from the Project Gutenberg collection. The package includes tools both for downloading books (stripping out the unhelpful header/footer information), and a complete dataset of Project Gutenberg metadata that can be used to find works of interest. In this book, we will mostly use the function gutenberg_download() that downloads one or more works from Project Gutenberg by ID, but you can also use other functions to explore metadata, pair Gutenberg ID with title, author, language, etc., or gather information about authors.

Let see how many authors and subjects in Gutenberg collection?

library(gutenbergr)
#List first 10 authors
gutenberg_authors$author[1:10]

#List first 10 subjects:
gutenberg_subjects$subject[1:10]

Create the collections of books by author Jane Austen

books <- gutenberg_works(author == "Austen, Jane")
dim(books)

A tibble is a modern class of data frame within R, available in the dplyr and tibble packages, that has a convenient print method, will not convert strings to factors, and does not use row names. Tibbles are great for use with tidy tools.

The dimension of books is (10,8) with 10 rows and 8 cols.

Each col is a meta data:

colnames(books)
print(books)

Each book title is has its own id and we can access the content of each book via downloading its id

book_105_121 <- gutenberg_download(c(105,121))

Now we can see that the tible book_105_121 is created and it has shape of 16319 rows with 2 cols: id and text. The IDs have only 2 values 105 and 121 and the text is the content of the selected book’s ID

print(dim(book_105_121))
head(book_105_121$text,20)

Now we will go into detail of text mining for this book by Jane Austen.

For simplicity, we use only book id 105 for our analyses:

book_105 <- gutenberg_download(105)

Convert the tible data to Corpus format for use in tm library

mydc <- VCorpus(VectorSource(book_105$text))

Manipulate the data

mydc <- tm_map(mydc,content_transformer(tolower))
mydc <- tm_map(mydc,removePunctuation)
mydc <- tm_map(mydc,removeWords,stopwords("english"))
mydc <- tm_map(mydc,stripWhitespace)
mydc <- tm_map(mydc,PlainTextDocument)
mydc_lemmatize <- tm_map(mydc,content_transformer(lemmatize_strings))

Calculte term frequency on the Corpus data

mydc_tdm <- TermDocumentMatrix(mydc)
CorpusMatrix <- as.matrix(mydc_tdm)
sortedMatrix <- sort(rowSums(CorpusMatrix))
dfCorpus <- data.frame(word = names(sortedMatrix),freq=sortedMatrix)
wordcloud2(data=dfCorpus,size=1.6)

library(RWeka)
dfNgrams <- data.frame(text=sapply(mydc,as.character),stringsAsFactors = FALSE)
uniGramToken <- NGramTokenizer(dfNgrams,Weka_control(min=1,max=1))
biGramToken <- NGramTokenizer(dfNgrams,Weka_control(min=2,max=2))
triGramToken <- NGramTokenizer(dfNgrams,Weka_control(min=3,max=3))

uniGrams <- data.frame(table(uniGramToken))
biGrams  <- data.frame(table(biGramToken))
triGrams <- data.frame(table(triGramToken))

uniGrams <- uniGrams[order(uniGrams$Freq,decreasing=TRUE),]
colnames(uniGrams) <- c('Word','Frequency')
biGrams <- biGrams[order(biGrams$Freq,decreasing=TRUE),]
colnames(biGrams) <- c('Word','Frequency')
triGrams <- triGrams[order(triGrams$Freq,decreasing=TRUE),]
colnames(triGrams) <- c('Word','Frequency')

plotUniGrams <- ggplot(head(uniGrams,10),aes(x=Frequency,y=reorder(Word,Frequency),fill=Word))+
                  geom_bar(stat='identity')+
                  scale_fill_brewer(palette="Spectral")+
                  geom_text(aes(x=Frequency,label=Frequency,vjust=1))+
                  labs(x="Frequency (%)",y="Words",title="UniGrams Frequency")
                  
                  
plotUniGrams

plotBiGrams <- ggplot(head(biGrams,10),aes(x=Frequency,y=reorder(Word,Frequency),fill=Word))+
                  geom_bar(stat='identity')+
                  scale_fill_brewer(palette="Spectral")+
                  geom_text(aes(x=Frequency,label=Frequency,vjust=1))+
                  labs(x="Frequency (%)",y="Words",title="BiGrams Frequency")
                  
                  
plotBiGrams

plotTriGrams <- ggplot(head(triGrams,10),aes(x=Frequency,y=reorder(Word,Frequency),fill=Word))+
                  geom_bar(stat='identity')+
                  scale_fill_brewer(palette="Spectral")+
                  geom_text(aes(x=Frequency,label=Frequency,vjust=1))+
                  labs(x="Frequency (%)",y="Words",title="TriGrams Frequency")
                  
                  
plotTriGrams

wordcloud2(uniGrams,size=1.6)

wordcloud2(biGrams,size=1.6)

wordcloud2(triGrams,size=1.6)

Key Points

  • Text Mining, R

Part of Speech Tagging

Overview

Teaching: 20 min
Exercises: 0 min
Questions

  • How to Perform POS using R

Objectives

  • Learn POS technique with R

12 Part of Speech Tagging

Here we use R OpenNLP to perform Part of Speech Tagging

The POS tags each token with their corresponding parts of speech, using statistics, context, meaning and their relative position with respect to adjacent tokens.

Beside existing CRAN package NLP, openNLP we need to install other library openNLPmodels.en:

install.packages("openNLPmodels.en", repos = "http://datacube.wu.ac.at/", type = "source")

Load neccesary library

Make sure all libraries are install prior to loading:

library(NLP)
library(openNLP)
library(openNLPmodels.en)
library(dplyr)
library(stringr)
library(ggplot2)

Load dataset

Here we use the dataset Gutenberg introduced in the previous step:

We use the book 105 from author “Austen, Jane” and define str1 as the content of the book

library(gutenbergr)
books <- gutenberg_works(author == "Austen, Jane")
book_105 <- gutenberg_download(105)
head(book_105$text,40)
str1 = as.String(book_105$text)

 [1] "Persuasion"      ""                "by Jane Austen"  ""               
 [5] "(1818)"          ""                ""                "Contents"       
 [9] ""                " CHAPTER I."     " CHAPTER II."    " CHAPTER III."  
[13] " CHAPTER IV."    " CHAPTER V."     " CHAPTER VI."    " CHAPTER VII."  
[17] " CHAPTER VIII."  " CHAPTER IX."    " CHAPTER X."     " CHAPTER XI."   
[21] " CHAPTER XII."   " CHAPTER XIII."  " CHAPTER XIV."   " CHAPTER XV."   
[25] " CHAPTER XVI."   " CHAPTER XVII."  " CHAPTER XVIII." " CHAPTER XIX."  
[29] " CHAPTER XX."    " CHAPTER XXI."   " CHAPTER XXII."  " CHAPTER XXIII."
[33] " CHAPTER XXIV."  ""                ""                ""               
[37] ""                "CHAPTER I."      ""                ""

Apply POS to the dataset

init_s = annotate(str1, list(Maxent_Sent_Token_Annotator(),
                               Maxent_Word_Token_Annotator()))
pos_res = annotate(str1, Maxent_POS_Tag_Annotator(), init_s)
word_subset = subset(pos_res, type=='word')
tags = sapply(word_subset$features , '[[', "POS")

pos1 = data_frame(word=str1[word_subset], pos=tags) %>% 
  filter(!str_detect(pos, pattern='[[:punct:]]'))
head(pos1,10)


# A tibble: 10 x 2
   word       pos  
   <chr>      <chr>
 1 Persuasion NNP  
 2 by         IN   
 3 Jane       NNP  
 4 Austen     NNP  
 5 1818       CD   
 6 Contents   NNPS 
 7 CHAPTER    NNP  
 8 I.         NNP  
 9 CHAPTER    NNP  
10 II         NNP

Plotting POS output

df1 = pos1 %>% 
      group_by(pos) %>% 
      summarise(n=n()) %>%
      mutate(freq=n/sum(n)) %>%
      arrange(desc(freq)*100)
      
ggplot(data=df1, aes(x=freq, y=pos,fill=pos)) +
  geom_bar(stat="identity")

image

Key Points

  • NLP, POS