Determining the Success Rate of Predictive Analytics in E-commerce: A Comprehensive Guide

When it comes to leveraging predictive analytics in e-commerce, the success rate is a critical metric that both companies and analysts must understand. This article aims to provide a deep dive into the success criteria for e-commerce applications, reflecting on the insights from Chapter 2 and Chapter 9 of Applied Predictive Analytics by Wiley.

Understanding Success Criteria in E-commerce

The determination of what is considered a good model depends on the specific business objectives of the organization. This is known as the business success criterion, which needs to be transformed into a predictive modeling criterion for an analyst to use in selecting models.

Success Criteria for Classification

In the context of e-commerce, classification models are often used to predict customer behavior, such as whether a user will purchase a product, abandon their cart, or engage in upselling. The primary metrics for assessing the accuracy of classification models are the Percent Correct Classification (PCC) and confusion matrices, which provide a summary of different types of errors such as Type I and Type II errors, precision, and recall.

For classification tasks, a high PCC score is desirable, but it must be measured in the context of the business objective. For instance, if a model's primary goal is to serve customized content based on browsing behavior, every visitor needs a model score and a custom treatment. On the other hand, if the objective is to isolate a specific subset of users, metrics such as lift, gain, and ROC curves become more relevant. The Area Under the Curve (AUC) is particularly popular in scenarios where a sub-population needs to be targeted with marketing messages or for fraud detection.

Success Criteria for Estimation

In the e-commerce realm, continuous-valued estimation problems often involve predicting metrics like customer lifetime value (CLV), sales forecasts, or estimating the impact of various marketing strategies. For these tasks, metrics like R2, Mean Squared Error (MSE), and average absolute error (AAE) are commonly used to evaluate the accuracy of the models.

These metrics are calculated by first computing the error of an estimate (actual value minus predicted estimate), and then the appropriate statistic based on those errors, which are then summed over all the records in the data. These metrics are useful in determining whether the models are biased and in estimating the magnitude of the errors.

Customized Success Criteria

In some cases, the typical success criteria may not suffice to evaluate predictive models because they do not align with the specific business needs. For instance, in e-commerce, the goal might be to identify the top 100 invoices for investigation from hundreds of thousands of invoices submitted. Here, a model that maximizes PCC might not be effective if the model fails to identify the most critical invoices.

To address such situations, customized cost functions can be used. These functions take into account the specific costs and benefits associated with different outcomes. For example, in fraud detection, a false alarm (investigating a non-fraudulent invoice) might be more costly than not detecting a true fraud. Therefore, the model should be chosen to minimize false alarms and maximize true alerts, with a penalty for false alarms that reflects the cost of investigation versus the gain from successful fraud recovery.

Choosing the Right Measurement

Choosing the right measurement can significantly impact the performance of the model. It is crucial to understand the business objective and select the appropriate metric. For example, if the objective is to select one-third of the population for treatment, model gain or lift at the 33 percent depth is appropriate. Conversely, if the objective is to maximize selects subject to a maximum false alarm rate, a ROC curve is appropriate.

Using the wrong metric can lead to sub-optimal results. For instance, a model ranked highly by AUC at a specific depth might not be the best choice if the primary objective is to minimize mean squared error. Therefore, it is essential to ensure that the metric used matches the business objectives, and the model selected based on one set of metrics might not perform well under another.

Conclusion

The success rate of predictive analytics in e-commerce is not just about the accuracy of the model. It is about aligning the metrics used for evaluation with the specific business objectives. By understanding the business needs and selecting the appropriate success criteria, analysts can build models that deliver the best possible results for their clients.

About the Author

Dean Abbott is the Co-Founder and Chief Data Scientist of Smarter Remarketer Inc. and President of Abbott Analytics Inc. in San Diego, California. Mr. Abbott is an internationally recognized data mining and predictive analytics expert with over two decades of experience applying advanced data mining algorithms, data preparation techniques, and data visualization methods to real-world problems, including fraud detection, risk modeling, text mining, personality assessment, response modeling, survey analysis, planned giving, and predictive toxicology.