Evaluation Metrics for Regression models- MAE Vs MSE Vs RMSE vs RMSLE

If you are working on a regression-based machine learning model like linear regression, one of the most important tasks is to select an appropriate evaluation metric.

In fact, if you are working on a machine learning projects in general or preparing to become a data scientist, it’s kind of must for you to know the top evaluation metrics.

These are also called loss functions.

There are two kinds of machine learning problems – classification and regression.

And these have different kind of loss functions.

In this post, I am going to talk about regression’s loss functions.

Since every project or data set is different, we must select appropriate evaluation metrics. Usually, more than 1 metrics is required to evaluate a machine learning model.  

Instead of including all the loss functions or evaluation metrics for regression machine learning models, I will try to focus on top loss functions. 

Download android app for better experience.

Before we start with loss functions, you need to understand what we are trying to do here. In a typical regression-based machine learning model, our model will produce continuous values (predicted value).

Our primary objective is to keep these predicted values closer to actual values.

Predicted values are denoted by y hat ().

Actual values are denoted by y.

Error = y - y hat

So, whenever we are talking about error in this post, we are talking about this error. And yes, ideal condition (hypothetical one) is that this error (difference) is 0, which means our model can predict all values correctly (which is not going to happen).

Let’s start with mean absolute error.

Mean absolute error (MAE)

In simple terms, mean absolute error is the sum of absolute/positive errors of all values. So, if there are 5 values in our data set, we find out the difference between the actual value and predicted values for all 5 values and take their positive value. So even if the difference between actual and predicted value is negative, we take positive value for calculation.

So we take the positive value of all errors, add them and find out their mean.

Mean absolute error illustration;

MAE is the sum of absolute differences between actual and predicted values. It doesn’t consider the direction, that is, positive or negative.

When we consider directions also, that is called Mean Bias Error (MBE), which is a sum of errors(difference).  

Formula for mean absolute Error or MAE is represented by;

Mean Square Error (MSE)

Mean square error is always positive and a value closer to 0 or a lower value is better. Let’s see how this this is calculated;

Let’s use the last illustration to understand it better.

So if we were to run a model with different parameters/independent variables, model with lower MSE will be deemed better.

We will look at its comparison with other loss functions in a while in this post. First quickly cover RMSE.

Root mean square error (RMSE)

Square root of MSE yields root mean square error (RMSE). So it’s formula is quite similar to what you have seen with mean square error, it’s just that we need to add a square root sign to it;

It is the standard deviation of error (residual error).

it indicates the spread of the residual errors. It is always positive, and a lower value indicates better performance. Ideal value would be 0 but it is never achieved.

Effect of each error on RMSE is directly proportional to the squared error therefore, RMSE is sensitive to outliers and can exaggerate results if there are outliers in the data set.

Before moving to their comparison, I just want to mention one more evaluation metric and that is Root mean squared log error (RMSLE)

Root mean squared log error (RMSLE)

Root mean squared log error is basically RMSE but calculated at logarithmic scale. So, if you understand the above mentioned 3 evaluation metrics, you won’t have any problem understanding RMSLE or most other evaluation metric or loss functions used in regression-based machine learning model.

While calculating RMSLE, 1 is added as constant to actual and predicted values because they can be 0 and log of 0 is undefined. Overall formula remains same. Standard denotation for RMSLE is;

In this illustration, I have used log for calculation;

Let’s look their difference now.

MAE vs MSE vs RMSE Vs RMSLE

In terms of comparison, primary differences are between MAE & MSE because they both are calculated in different ways. RMSE & RMSLE are extension of MSE therefore they share lots of properties with MSE.

MAE vs MSE vs RMSE Vs RMSLE Conclusion

I have mentioned only important differences. If there is no valid point for one, I haven’t included in the above table and that’s why we have empty cells in the table.

Few important points to remember when using loss functions for your regression;

• Never compare apple with oranges, that is, never compare different metrics with each other. For example- don’t compare values of MSE with MAE or others. They would be different.
• Try to use more than 1 loss function.
• Always calculate evaluation metrics (loss functions) for both testing and training data set.
• Compare evaluation metrics between test and training data set. There shouldn’t be a huge difference between them. If there is, there is a problem with your model. For example- if you are using RMSE, calculate RMSE for testing and training data set. There should be huge difference between these values for this data set.
• If you have outlier in the data and you want to ignore them, MAE is a better option but if you want to account for them in your loss function, go for MSE/RMSE. 

Questions or feedback? Please leave your comments.