Detecting Fraud in Credit Card Data

library(tidyverse)
library(mlr)
library(parallelMap)

Introduction

The objective of this analysis is to show how one can use the amazing tools available in R to create a model and take it into a production environment. The dataset used comes from Kaggle (https://www.kaggle.com/mlg-ulb/creditcardfraud/) and contains a number of transactions done by credit card and some of them are flagged as fraudulent.

The data

The data has been modified before made available, most of the features have been compressed into 28 variables probably using a PCA algorithm. That leaves us with 28 unnamed variables v1:v28 and 3 more variables, Time, Amount and Class.

Class is our target variable where 0 means not fraud and 1 means fraud. The feature time only contains the seconds elapsed in between transactions most likely is not useful for our model creation.

df <- read_csv("./input/creditcard.csv", progress = FALSE, col_types = cols())
df %>% head()
df$Class <- factor(df$Class)
df <- data.frame(df)

The dataset is highly unbalanced with only a small percentage of the observations being fraudulent. Less than 0.2% of the transactions are fraudulent. This makes it tricky to effectively split the dataset into two.

table(df$Class)
prop.table(table(df$Class))

At this stage we divide the data into train and test datasets. One is used for model calibration and the other for validation.

Model selection

A boosted model will be used in this analysis, a more in depth look at models available would be recommended however this is more of a high level exercise so we will assume that a decision has been made of using a Gradient Boosting Machine (GBM) that should give us decent prediction results as in general it deals well with imbalanced data. We will use the Extreme Boosting (XGBM) implementations as it provides better performance and shorter training times.

Feature engineering

Again the PCA has made the possibility of performing feature engineering pointless, the data is already heavily modified. For the purpose of this example we will continue with the data as is.

Building the model

We are using mlr for our modelling needs as it makes it easy to perform hyper-parameter tuning and cross validation to improve the quality of our model. MLR is a framework that unifies different outputs from different machine learning algorithms unifying the interface and providing tools for analysis and model optimization.

We split the dataset with a 70 - 30 train test split. We verify then that the proportion is roughly the same given the low number of fraudulent transactions.

set.seed(42)

train.test <- sample(2 # either one or two
    , nrow(df)
    , replace = TRUE
    , prob = c(0.7, 0.3))
df_train = df[train.test == 1,]
df_test = df[train.test == 2,]

prop.table(table(df_train$Class))
prop.table(table(df_test$Class))

# We will use 70% of the observations for training

# Make the task
class.task <- makeClassifTask(data = df_train, target = "Class")

# Make a learner
#gbm.learn <- makeLearner("classif.gbm")
xgbm.learn <- makeLearner("classif.xgboost")

A run with the default parameters will render the following: That is a decent result we only have one false negative i.e. a fraudulent transaction that is considered good and we have flagged 144 transactions as fraud when they were good.

While not perfect we can see that GBM can deal with the imbalance in the data.

model <- train(learner = xgbm.learn, task = class.task)
performance(predict(model,newdata = df_test), measures = list(mmce, ber, acc, fn, fp, fnr,fpr, bac))

#getParamSet(gbm.learn)
getParamSet(xgbm.learn)

For these tasks we are going to define a random search. This paper [http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf] lays out why random search is recommended in general as you will spend less time searching and it’s guaranteed according to the authors to give better results given the same amount of iterations. For a quick sweep where you want to test few parameters

MLR supports more advanced parameter tuning functions although they are more difficult to run in parallel.

Our best results seem to come from a tree depth of 12 a subsample of 0.967, a colsample of 0.955 and a learning rate of 0.794.

The new model with the best parameters achieves a better result in every metric than before the results on the df_test dataset can be seen below. It could be refined to avoid false negatives altogether however that will depend on our business objectives and this results seem a good compromise.

lrn_best <- setHyperPars(xgbm.learn, par.vals = xgbm_tune$x)

model_best <- train(lrn_best,class.task)
performance(predict(model_best,newdata = df_test), measures = list(mmce, ber, acc, fn, fp, fnr,fpr, bac))

Now that we have our model how can we use it to make predictions? Below we see how a set of parameters called example can be passed to our model. In this case we are passing a 1 observation dataframe.

example <- df_test["6335",1:(ncol(df_test))]
example

res <- predict(model_best, newdata = example)
res <- res$data$response %>% as.character() %>% as.numeric()
res

The result is 1 for fraud which is correct in this case. However being able to predict fraud on old data is not very useful, for this model to add real value we need to be able to connect it to our existing services so we can obtain fast results and cancel the fraudulent transactions for our clients.

To do that we need to save the results on a rds file (or we pickle the model in Python lingo):

saveRDS(model_best,"./output/model_best.rds")

See More on Part 2: Deploying a model