Exercise - Data manipulation part 3 - Impute missing values by using machine learning

• 8 minutes

As player_df.isna().sum() confirmed in the previous unit, only nine missing values remain in PER. You can't use a simple average to impute values in that column. But little domain expertise can help.

PER is computed from the values of the nine columns that precede it in the DataFrame (GP through REBR). However, PER is a complicated statistic to compute (see details at basketball-reference.com). So you'll do what data scientists do: build a model to give a good approximation!

It would be nice to build a simple linear regression model to estimate the missing PER values. However, as with any estimate, you have to ask yourself how accurate it is.

To get some sense of a model's accuracy, you could use machine learning to split your data into two subsets: test and training. The training subset is the portion of the data you use to train the model. You use the other subset to test the model. Commonly, 75 percent of the data is used to train the model, and 25 percent is used to test the model.

But what if you get unusually lucky or unlucky with your random split of test and training data? For example, what if you chose all human players to train the model, but then you tested the accuracy of the model on the Tune Squad players? This setup is unlikely to yield accurate results.

To avoid this problem, statisticians and data scientists use a technique called cross-validation. The idea is to iterate through the dataset, splitting the data in different ways between training data and test data. Using this technique multiple times should give you a reasonable idea of how the model will work with new data, even if you have only limited data to work with.

This image provides a visualization of the cross-validation process:

Cross-validate the R2 scores for the model

Here you'll use a 10-fold cross validation. That is, Python will iterate through the data 10 times, reserving 10 percent of the data for testing and training on the other 90 percent of the data each time. You'll also plot a histogram of the results.

The code of the predicted value (y) is similar to the code for X, except that you don't have to scrape out just the values.

from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
from sklearn.model_selection import cross_val_score

# Define the variables for the regression model as those rows that have no missing values.
X = player_df.dropna(how='any').iloc[:, 5:-1].to_numpy()
y = player_df.dropna(how='any').iloc[:, -1]

# Define the regression model.
lin_reg = LinearRegression()

# Use the scikit-learn cross-validation function to fit this model 10 times and return the R2 scores.
scores = cross_val_score(lin_reg, X, y, cv=10, scoring='r2')

# Define the histogram of the scores and copy out information from the histogram.
entries, bin_edges, patches = plt.hist(scores, bins=10);

# Print out the mean and the results from the histogram.
print('Mean r2 score: {:.4f}'.format(scores.mean()))
for i in range(len(entries)):
    if entries[i] > 0:
        print('{:.0f}% of r2 scores are between {:.4f} and {:.4f}'.format(entries[i]*100/len(entries), 
        bin_edges[i], 
        bin_edges[i+1]))

Mean r2 score: 0.9995
10% of r2 scores are between 0.9985 and 0.9987
10% of r2 scores are between 0.9987 and 0.9988
10% of r2 scores are between 0.9995 and 0.9996
10% of r2 scores are between 0.9996 and 0.9997
60% of r2 scores are between 0.9997 and 0.9998

In short, your model is good. Your R2 score is 99.95 percent. The R2 (meaning, R squared) score tells you how much data variance your model captures. For your purposes, this score serves as a loose proxy for the model's accuracy. Your lowest R2 scores are still good, and most of them are close to the maximum value of 1. So you should feel confident applying this model to your missing PER values.

Fit the regression model for the player data

Now you fit the regression model to all of your data. (The general rule is that you use cross-validation for model selection or evaluation, but you use all of your data for model building.)

# Fit the same regression model, this time using all of the available data.
lin_reg.fit(X, y)

LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)

Create a mask of rows that use missing values in the DataFrame

There isn't an elegant way for fillna() to use the results from your model to fill in the missing PER values. So you'll have to use a mask.

A mask in pandas is a Boolean map of values that fulfill a certain condition or set of conditions. To fill in just your remaining values, your mask should look like this:

# Create and display a mask of rows in the DataFrame. Rows should contain at least one NaN value.
mask = player_df.isnull().any(axis=1)
mask

0     False
1     False
2      True
3     False
4     False
5     False
6     False
7      True
8      True
9     False
10    False
11     True
12    False
13    False
14    False
15     True
16    False
17    False
18    False
19    False
20     True
21    False
22    False
23    False
24     True
25    False
26    False
27    False
28    False
29     True
30    False
31    False
32    False
33    False
34    False
35    False
36    False
37     True
38    False
39    False
40    False
dtype: bool

Now apply that mask to just the columns that you used as X in your model. Use the loc DataFrame method:

# Apply the mask defined earlier to show the contents of specific columns of rows that contain NaN values.
player_df.loc[mask].iloc[:, 5:-1]

Output

Now you've identified the nine rows that have a NaN value in the PER column and have the columns that can be used in the machine learning model to predict the PER value.

Use the mask and the fitted mask to impute the final missing values in the DataFrame

A mask is useful here because it lets you use a view of the player_df DataFrame rather than a slice of it. When you defined X, you used player_df.dropna(how='any').iloc[:, 5:-1].to_numpy(). The dropna() method creates a new DataFrame object by default when it's called. (This feature is the reason you used the method's inplace parameter on the original DataFrame.) So any changes to values that you might make in the X DataFrame wouldn't change values in the player_df DataFrame.

Your mask is different. It simply enables you to work with a subset of the player_df DataFrame. So any changes you make to the DataFrame while you're applying the mask will also apply to the player_df DataFrame as a whole.

# Impute the missing values in 'PER' by using the regression model and mask.
player_df.loc[mask, 'PER'] = lin_reg.predict(player_df.loc[mask].iloc[:, 5:-1])

# Recheck the DataFrame for rows that have missing values.
player_df.isna().sum()

ID             0
player         0
points         0
possessions    0
team_pace      0
GP             0
MPG            0
TS%            0
AST            0
TO             0
USG            0
ORR            0
DRR            0
REBR           0
PER            0
dtype: int64

Next, review the entire DataFrame to check for any missing values.

# Display the entire DataFrame.
player_df

Output

Finally, save this DataFrame to a CSV file. You'll use this dataset to make pregame and in-game decisions based on the team players, the players who are currently in the game, and the current game stats.

player_df.to_csv('player_data_final.csv', index=False)

You should now see a new CSV file in your space-jam-anl folder.

© 2021 Warner Bros. Ent. All Rights Reserved.