1.1 Package presentation

Based on data.table package, dataPreparation will allow you to do most of the painful data preparation for a data science project with a minimum amount of code.

This package is

• fast (use data.table and exponential search)
• RAM efficient (perform operations by reference and column-wise to avoid copying data)
• stable (most exceptions are handled)
• verbose (log a lot)

data.table and other dependencies are handled at installation.

1.2 Main preparation steps

Before using any machine learning (ML) algorithm, one needs to prepare its data. Preparing a data set for a data science project can be long and tricky. The main steps are the followings:

• Read: load the data set (this package don’t treat this point: for csv we recommend data.table::fread)
• Correct: most of the times, there are some mistake after reading, wrong format… one have to correct them
• Transform: creating new features from date, categorical, character… in order to have information usable for a ML algorithm (aka: numeric or categorical)
• Filter: get read of useless information in order to speed up computation
• Handle NA: replace missing values
• Shape: put your data set in a nice shape usable by a ML algorithm

Here are the functions available in this package to tackle those issues:

All of those functions are integrated in the full pipeline function prepareSet.

In this tutorial we will detail all those steps and how to treat them with this package using an example data set.

1.3 Tutorial data

For this tutorial, we are going to use a messy version of adult data base.

data(messy_adult)
print(head(messy_adult, n = 4))

#         date1      date2       date3           date4    num1    num2
# 1: 2017-10-07         NA 19-Jan-2017 21-January-2017 -3.0953  0,4954
# 2: 2017-31-12 1513465200 06-Jun-2017    08-June-2017  0.2227 -0,8202
# 3: 2017-12-10 1511305200 03-Jul-2017    05-July-2017 -0.2916  -0,713
# 4: 2017-06-09 1485126000 19-Jul-2017    21-July-2017  2.3236  0,7155
#    constant                             mail    num3 age    type_employer
# 1:        1            marie.cynthia@aol.com -3,0953  39        State-gov
# 2:        1     jake.caroline@protonmail.com  0,2227  50 Self-emp-not-inc
# 3:        1 caroline.caroline@protonmail.com -0,2916  38          Private
# 4:        1         caroline.marie@yahoo.com  2,3236  53          Private
#    fnlwgt education education_num            marital        occupation
# 1:  77516 Bachelors            13      Never-married      Adm-clerical
# 2:  83311 Bachelors            13 Married-civ-spouse   Exec-managerial
# 3: 215646   HS-grad             9           Divorced Handlers-cleaners
# 4: 234721      11th             7 Married-civ-spouse Handlers-cleaners
#     relationship  race  sex capital_gain capital_loss hr_per_week
# 1: Not-in-family White Male         2174            0          40
# 2:       Husband White Male            0            0          13
# 3: Not-in-family White Male            0            0          40
# 4:       Husband Black Male            0            0          40
#          country income
# 1: United-States  <=50K
# 2: United-States  <=50K
# 3: United-States  <=50K
# 4: United-States  <=50K

We added 9 really ugly columns to the data set:

• 4 dates with various formats, or time stamps, and NAs
• 1 constant column
• 3 numeric with different decimal separator
• 1 email address

The same info can be contained in two different columns.

2.1 Identifying factor that shouldn’t be

It often happens when reading a data set that R put string into a factor even if it shouldn’t be. In this tutorial data set, mail is a factor but shouldn’t be. It will automatically be detected using unFactor function:

print(class(messy_adult$mail))

# "factor"

messy_adult <- unFactor(messy_adult)

# "unFactor: I will identify variable that are factor but shouldn't be."
# "unFactor: I unfactor mail."
# "unFactor: It took me 0.14s to unfactor 1 column(s)."

print(class(messy_adult$mail))

# "character"

2.2 Identifing and transforming date columns

The next thing to do is to identify columns that are dates (the first 4 ones) and transform them.

messy_adult <- findAndTransformDates(messy_adult)

# "findAndTransformDates: It took me 0.28s to identify formats"
# "findAndTransformDates: It took me 0.23s to transform 4 columns to a Date format."

As one can see, even if formats were different and somehow ugly, they were all handled.

2.3 Identifying and transforming numeric columns

And now the same thing with numeric

messy_adult <- findAndTransformNumerics(messy_adult)

# "findAndTransformNumerics: It took me 0.14s to identify 3 numerics column(s), i will set them as numerics"
# "setColAsNumeric: I will set some columns as numeric"
# "setColAsNumeric: I am doing the columnnum1"
# "setColAsNumeric: 0 NA have been created due to transformation to numeric."
# "setColAsNumeric: I will set some columns as numeric"
# "setColAsNumeric: I am doing the columnnum2"
# "setColAsNumeric: 0 NA have been created due to transformation to numeric."
# "setColAsNumeric: I am doing the columnnum3"
# "setColAsNumeric: 0 NA have been created due to transformation to numeric."
# "findAndTransformNumerics: It took me 0.06s to transform 3 column(s) to a numeric format."

So now our data set is a bit less ugly.

3.1 Look for constant variables

constant_cols <- whichAreConstant(messy_adult)

# "whichAreConstant: constant is constant."
# "whichAreConstant: it took me 0.17s to identify 1 constant column(s)"

3.2 Look for columns in double

double_cols <- whichAreInDouble(messy_adult)

# "whichAreInDouble: num3 is exactly equal to num1. I put it in drop list."
# "whichAreInDouble: it took me 0.19s to identify 1 column(s) to drop."

3.3 Look for columns that are bijections of one another

bijections_cols <- whichAreBijection(messy_adult)

# "whichAreBijection: date4 is a bijection of date3. I put it in drop list."
# "whichAreBijection: num3 is a bijection of num1. I put it in drop list."
# "whichAreBijection: education_num is a bijection of education. I put it in drop list."
# "whichAreBijection: it took me 0.47s to identify 3 column(s) to drop."

To control this, let’s have a look to the concerned columns:

kable(head(messy_adult[, .(constant, date3, date4, num1, num3, education, education_num)])) %>%
   kable_styling(bootstrap_options = c("striped", "hover"), full_width = FALSE, font_size = 12)

Indeed:

• constant was build constant, it contains only 1,
• num1 and num3 are equal,
• date3 and date4 are separated by 2 days: date4 doesn’t contain any new information for a ML algorithm,
• education and education_num contains the same information one with a key index, the other one with the character corresponding. whichAreBijection keeps the character column.

3.4 Filter them all

To directly filter all of them:

ncols <- ncol(messy_adult)
messy_adult <- fastFilterVariables(messy_adult)
print(paste0("messy_adult now have ", ncol(messy_adult), " columns; so ", ncols - ncol(messy_adult), " less than before."))

# "fastFilterVariables: I check for constant columns."
# "fastFilterVariables: I delete 1 constant column(s) in dataSet."
# "fastFilterVariables: I check for columns in double."
# "fastFilterVariables: I delete 1 column(s) that are in double in dataSet."
# "fastFilterVariables: I check for columns that are bijections of another column."
# "fastFilterVariables: I delete 2 column(s) that are bijections of another column in dataSet."
# "messy_adult now have 20 columns; so 4 less than before."

4 useless rows have been deleted. Without those useless columns, your machine learning algorithm will at least be faster and maybe give better results.

4.1 Dates differences

Since no machine learning algorithm handle Dates, one needs to transform them or drop them. A way to transform dates is to perform differences between every date.

We can also add an analysis date to compare dates with the date your data is from. For example, if you have a birth-date you may want to compute age by performing today - birth-date.

date_cols <- names(messy_adult)[sapply(messy_adult, is.POSIXct)]
messy_adult <- generateDateDiffs(messy_adult, cols = "auto", analysisDate = as.Date("2018-01-01"), units = "days")

# "generateDateDiffs: I will generate difference between dates."
# "generateDateDiffs: It took me 0.15s to create 6 column(s)."

4.2 Transforming dates into aggregates

Another way to work around dates would be to aggregate them at some level. This time drop is set to TRUE in order to drop date columns.

messy_adult <- generateFactorFromDate(messy_adult, cols = date_cols, type = "quarter", drop = TRUE)

# "generateFactorFromDate: I will create a factor column from each date column."
# "generateFactorFromDate: It took me 0.19s to transform 3 column(s)."

4.3 Generate features from character columns

Character columns are not handled by any machine learning algorithm, one should transform them. Function generateFromCharacter build some new feature from them, and then drop them.

messy_adult <- generateFromCharacter(messy_adult, cols = "auto", drop = TRUE)

# "generateFromCharacter: it took me: 0s to transform 1 character columns into, 3 new columns."

4.4 Aggregate according to a key

To model something by country; one would want to to compute an aggregation of this table in order to have one line per country.

agg_adult <- aggregateByKey(messy_adult, key = "country")

# "aggregateByKey: I start to aggregate"
# "aggregateByKey: 139 columns have been constructed. It took 0.62 seconds. "

Every time you have more than one line per individual this function would be pretty cool.

4.5 Rounding

One might want to round numeric variables in order to save some RAM, or for algorithmic reasons:

messy_adult <- fastRound(messy_adult, digits = 2)