Team strength MLP (part 1)

It is time to build and test my first predictive model with Tensorflow! As I am currently totally unexperienced in creating and optimizing neural networks, I will start with a very simple one, which just uses the predictive variables of the Poisson model. By doing this, I will be able to compare the resulting network with the Poisson model. I am excited to see, whether Tensorflow is able to outperform this statistical model with such a low number of predictive variables. In this series I will provide some basic information, how you are able to build a simple multilayer perceptron (MLP) with Tensorflow, supervise the training process with Tensorboard and use the trained neural network to predict the outcomes of the matches.

What is a MLP?

A multilayer perceptron is one class of deep, artificial neural networks, which consists perceptrons. A perceptron is the artificial neuron of a neural network. And as the name suggests, a MLP consists of multiple layers. The predictive variable or signals are provided throw the input layer. The output layer provides the prediction based on the input. Between both, at minimum one, hidden layers are used.

A MLP belongs to the class of supervised learning models. A MLP is trained by a set of observations and known results. This is achieved by a MLP as it uses back-propagation. All the information flows throw the network and generates a result. Based on the comparison of these output values and the correct output, the back-propagation algorithm adjusts the weights of the single neurons. So the neural network learns the relation between the input and the output values.

At one site I read, that a MLP is the “hello world of deep learning”. So this is a got starting point to learn, how neural networks and deep learning works.

Prerequisites

Using the Tensorflow framework directly is hard work. To get an easy start I decided to use a higher-level framework. There are currently two popular APIs: TFLearn and Keras. I decided to use TFLearn, but not because of any specific advantages, just because I found some nice code examples, which were easy to understand. And, if I got the community correct, none of the frameworks has got huge disadvantages.

So before I started, I had to install the TFLearn package to the existing Annaconda environment:

(<span class="mceItemHidden"><span class="hiddenSpellError"><span class="mceItemHidden"><span class="mceItemHidden"><span class="hiddenSpellError">tensorflow</span></span></span></span></span>)C:&gt; pip install <span class="hiddenSpellError"><span class="mceItemHidden"><span class="mceItemHidden"><span class="hiddenSpellError">tflearn</span></span></span></span>

Additionally I installed the package Sklear, which offers a simple function to split a dataset for test and validation:

(<span class="mceItemHidden"><span class="hiddenSpellError"><span class="mceItemHidden"><span class="mceItemHidden"><span class="hiddenSpellError">tensorflow</span></span></span></span></span>)C:&gt; pip install <span class="hiddenSpellError"><span class="mceItemHidden"><span class="mceItemHidden"><span class="hiddenSpellError">sklearn</span></span></span></span>

Extract training data

I already described, how you are able to connect to an Exasol database and extract data. So you just need to query the correct training data. I used the Bundesliga seasons 2010 – 2016 for training. The last season 2017/18 I would like to use for prediction and comparison to the Poisson model.

Following SQL statement provides all predictor variables and the full-time results for every match:

#read train data
df_data = v_Con.export_to_pandas("""
select
   str.home_attacking_strength,
   str.home_defence_strength,
   str.away_attacking_strength,
   str.away_defence_strength,
    case when stat.full_time_result = 'H' then 1 else 0 end home_win,
    case when stat.full_time_result = 'D' then 1 else 0 end draw,
    case when stat.full_time_result = 'A' then 1 else 0 end away_win
from
   betting_dv.football_match_his_l his
   join betting_dv.football_match_his_l_s_statistic stat
      on his.football_match_his_lid = stat.football_match_his_lid
   join betting_dv.football_match_his_l_s_attack_defence_strength_30 str
      on his.football_match_his_lid = str.football_match_his_lid
   join betting_dv.football_division_h division
      on his.football_division_hid = division.football_division_hid
   join betting_dv.football_season_h season
      on his.football_season_hid = season.football_season_hid
   join betting_dv.football_team_h home
      on his.football_team_home_hid = home.football_team_hid
   join betting_dv.football_team_h away
      on his.football_team_away_hid = away.football_team_hid
where
   division.division = '<span class="mceItemHidden"><span class="hiddenSpellError"><span class="mceItemHidden"><span class="mceItemHidden"><span class="hiddenSpellError">D1</span></span></span></span></span>' and
   season.season in ('2016_2017','2015_2016','2014_2015','2013_2014','2012_2013','2011_2012','2010_2011')
                                    """)

If you follow my posts, this statement should look familiar to you. The labels for the outcome of a match are transposed to single columns, so that every outcome class (home-win, draw, away-win) is represented with a single column. This is needed for the classification process with Tensorflow.

Build train and validation set

The data which is available in a pandas data frame has now to be split twice. At first you have to split the predictors and the labels, which you want to predict. This is needed for Tensorflow, as it expects separate labels.

Following statement splits the data frame:

#split predictors and outcome
df_x = df_data[df_data.columns[0:4]]
df_y = df_data[df_data.columns[4:7]]

In a second step both data frames have to be divided into a training and a validation set. During the training process of a neural network with Tensorflow, the training set is, as the name suggest, used to train the model. The back-propagation algorithm compares the result of the network with the provided labels and adjust the weights. With help of the validation set, the learn process can be monitored. The prediction result is just compared to the labels and the accuracy is checked. So you are aware of how the current training status of the network reacts to unknown data. Splitting the data can easily be achieved by the train_test_split() function of the SKlearn package. You just need to define the ration between the training and validation data.

#split train and validation data set
df_x_train, df_x_val, df_y_train, df_y_val = train_test_split(df_x, df_y, test_size=0.2, random_state=42)

I used 10% of the data set for validation, the rest for training. When you want to test and compare different models, you should also define the random_state. This ensures, that you get the same splitting each time during model testing.

Define MLP architecture

After all the data is prepared, you can start building the architecture of your neural network. By using TFLearn this is again fairly simple. My first model consists just of 3 layers – the input layer, the output layer and one hidden layer.

The input layer has to include 4 input neurons. One for each team strength variable. For the hidden layer I defined 10 neurons. The output layer contains 3 neurons, one for each match outcome label (home-win, draw, away-win). This is one of the most basic MLP architectures possible.

#build layers
#input layer 4 inputs
#3 hidden layers each 30 neurons
#output layer 3 outputs
v_net = tflearn.input_data(shape=[None, 4], name="InputLayer")

v_net = tflearn.fully_connected(v_net, 10, activation="relu", bias=False,
weights_init="truncated_normal",
bias_init="zeros",
trainable=True,
restore=True,
reuse=False,
scope=None,
name="FullyConnectedLayer_relu")

v_net = tflearn.fully_connected(v_net, 3,
activation="softmax",
name="OutputLayer")

v_net = tflearn.regression(v_net,
optimizer=tflearn.Adam(learning_rate=0.01, epsilon=0.01),
metric=tflearn.metrics.accuracy(),
loss="categorical_crossentropy"
)

#define model
model = tflearn.DNN(v_net,
tensorboard_dir='c:/temp/<span class="mceItemHidden"><span class="hiddenSpellError"><span class="mceItemHidden"><span class="mceItemHidden"><span class="hiddenSpellError">tensorflow</span></span></span></span></span><span class="mceItemHidden">/output/logs/',
best_val_accuracy=0.50,
best_checkpoint_path='c:/temp/<span class="mceItemHidden"><span class="hiddenSpellError">tensorflow</span></span>/output/</span><span class="hiddenSpellError"><span class="mceItemHidden"><span class="mceItemHidden"><span class="hiddenSpellError">ckpt</span></span></span></span>/',
tensorboard_verbose=0)

For the definition of the different layers you need several parameters, which need to be optimized during the training of a model. The most important ones are:

• activation function
• optimizer
• learning rate
• (momentum – not for Adam optimizer)

ReLu and Softmax activation functions are my choice for the beginning. ReLu is one of the most used activation functions and suggested as a starting point, when creating a new neural network. Softmax is the only possibility for the output layer. A football match can only have one outcome and the Softmax activation ensures this. The optimizer and the learning rate were just determined by some testing. The complete network and parameter optimization is of course a step, which should be automated. Otherwise you need a lot of time, to find the best parameters.

The next post will describe, how you train this model and how you can use Tensorboard to monitor this training process.

The complete source code can again be found at GitHub:

GitHub – Team strength MLP training

 

 

If you have further questions, feel free to leave a comment or contact me @Mo_Nbg.

 

 

References:

[1] Jason Brownlee, Crash Course On Multi-Layer Perceptron Neural Networks, 2016, https://machinelearningmastery.com/neural-networks-crash-course/

[2] Anish Singh Walia, Activation functions and it’s types-Which is better?, 2017, https://towardsdatascience.com/activation-functions-and-its-types-which-is-better-a9a5310cc8f