Perceived quality issues in forecasting
Software quality is a challenge. As a software developer, you have to make sure that your code is going to run in all sort of unexpected situations, but still it has to work. Plenty of methods, tools and processes are available to improve quality. Lokad makes an intensive usage of those.
But there is another aspect, it’s the perceived quality: the user’s opinion depends on many (many) purely subjective aspects of your product. For B2C, product design and aesthetic are probably among the top factors in perceived quality (think iPod).
So far, so good, as a product developer, it simply means that you need to invest a certain amount of efforts in your product design. But what happens when perceived quality conflicts actual quality? (think to the devil’s method to change your iPod battery).
In the case of Lokad, where we are delivering time-series forecasts, the situation is even more complicated because statistical forecasting is just so not intuitive.
For example, we have many customers who actually try out a couple of points to see what they get. Yet, this is really not the way to go to evaluate Lokad. The right way involves a proper training dataset and a testing dataset of your own actual business data (plus many other considerations, but it’s beyond the scope of this post).
Unfortunately, for us, many customers are judging Lokad on the forecast they get after entering a dozen of points generated by some function like Cos(x) or Sin(x). Actually, it would be possible to hard-code a few heuristics in Lokad just to detect those attempts (and their underlying mathematical functions). But, by doing so those heuristics would actually decrease the overall accuracy for the users having real business data in their accounts.
Then, we have another issue: our forecasts are not exactly real time. You can retrieve your forecasts any time, but if you retrieve your forecasts 0.1s after finishing the upload of your data (through our Web Services API), Lokad won’t have had the time to try complex/advanced statistical models. As a result, you will get real-time but naive forecasts.
Lokad does its best to provide an end-to-end forecasting service, but to some extend it can’t escape the Law of Leaky Abstraction: in order to make the most of Lokad, one needs to understand, at least little, how statistical forecasting works and how the constraints are handled by Lokad.
Reader Comments (2)
Hi Oliver, We have indeed though of providing all sort of indicators along the forecasts for a long time. Yet, this solution raises even more issues that it solves; because of the very same statistical learning intricacies. Thanks for pointing this out though. I think I will a proper post on the subject later on. 10 years ago | Joannes Vermorel
Joannes, Perhaps one way out of your predicament is to give estimated quality feedback to the user each time they retrieve forecasts. I don’t know much about sales forecasting or your app, but for example you could detect that too few points were used, or that they appear to be generated from a function and show a warning with the forecasts that explains that the quality can’t be trusted. Also, if they pull forecasts before an in-depth analysis is performed, the nature of the analysis performed should be echoed to the user, along with the expect resulting quality. This will help your users understand what is going on and why. And, as you are well aware, that is key to having happy users. 10 years ago | Oliver
