As a result of the Corona crisis, the question always arises how this crisis affects the performance of the models created by our software. This question cannot be answered universally, since the Corona crisis has very different effects on certain industries and business models. In order to be able to estimate possible effects, various scenarios and their effects on the quality of the forecast are described below. A distinction is made between effects during the crisis and effects after the crisis. It also describes how so-called back tests can be used to quickly and easily determine the effects on the quality of the forecast.
Scenario 1: Homogeneous effect on the whole business
The business is largely homogeneously affected by the Corona crisis. This means that the purchasing behavior of all customers has changed to the same extent across all products and channels.
If this is the case, the absolute level of the conversion and sales forecast will deviate more from the actual values than usual. However, the models are still able to differentiate between good and bad customers. Thus, the best customers can still be selected for campaigns.
Scenario 2: heterogeneous effect on the whole business
The business is dependent on e.g. certain product groups or customer groups affected differently by the corona crisis.
In this case, the absolute level of the forecast will deviate more than usual from the actual values. In addition, the ability of the models to differentiate between good and bad customers is also impaired. How strong this effect is depends on the heterogeneity of the change in consumer behavior.
Scenario 1: Return to previous consumer behavior
After the turbulent phase of the Corona crisis, customers are returning to their consumer behavior and behave exactly as they did before the crisis.
Models that were created before the crisis can be used again as usual. If you create new models, the period of the corona crisis should be excluded for the modeling part so that no “wrong” patterns distort the training of the new models.
Scenario 2: Permanent change in consumer behavior
The Corona crisis leads to a permanent change in consumer behavior, because e.g. certain products are no longer in demand.
In this case, it must be checked to what extent the existing models can continue to be used.
All of the effects described here apply to the use of machine learning models as well as to the use of heuristics, e.g. RFM (Recency, Frequency, Monetary).
A so-called backtest can be used to check the effects of the Corona Crisis on the quality of the forecasts (accuracy of the forecasts and ability to differentiate between good and bad customers). If the backtest shows that the model's ability to differentiate is impaired, it is possible to reduce the negative effect on the quality of the forecast by dividing the model into several sub-models. The improvement of the model ensemble compared to the single model can also be tested again using back tests.
We are currently seeing the following effects across our customers:
Our conclusion based on our current observations is that in many cases the models continue to work well. In areas with significant changes it is often either possible to create models which, despite the given situation, allow an evaluation of the customers or enable new use cases. As before the Corona crisis, the identification of the right use case is the most important factor for success.
If you have any questions about predictive modeling during and after the Corona crisis, please contact us using the form below.