Forecasting Technology
Statistical forecasting is an old topic. Since the 19th century, retailers have been routinely relying on sales forecasts to optimize their inventory levels. Thus, companies have been doing forecasts long before the advent of computers; and clearly, the world did not wait for Lokad to start elaborating smart forecasting methods.
Check also our
Roadmap for 2010.
Yet, we believe that
Lokad is bringing something new to the forecasting field. We believe that our approach is simpler, cheaper but also more accurate compared to traditional methods. This page aims to give you an clear understanding of our technology.
Forecasting as a Service
Companies send their data to Lokad, and we send forecasts back. Once data is transferred to Lokad, your forecasts are computed in a fully automated manner. Producing a forecast does not require any manual intervention. This architecture guarantees that your forecasts will be delivered in a timely manner even if your schedule is tight.
Our technology is designed to deal with your data just the way it happens to be. With traditional forecasting tools, you end up first performing long and complicated data preparations – for example, to get rid of exceptional points – and later on, you are typically asked to provide expert insights about your data such as the presence or not of seasonality.
On the contrary, Lokad deals with all those steps on its own. A careful preparation, before sending data to Lokad, might improve the accuracy of the forecasts, but if you happen not to have the time or the expertise to do this, then we just accommodate your data, and deliver nonetheless the best forecasts we can produce.
Designed for 3rd party integration
To benefit from Lokad, companies need to send us their data and then to integrate our forecasts right into their business. This is why Lokad has been designed, right from the start, to facilitate 3rd party integration.
We expose a
Forecasting API - API standing for Application Programming Interface - that is to say a machine-to-machine communication protocol that relies on a ubiquitous standard named SOAP. This standard is supported by all major software companies such as Microsoft, IBM, SAP, Google, ... The Forecasting API enables a programmatic integration of Lokad into virtually any 3rd party software as long as an internet connection is available.
Also, in order to facilitate the integration of our technology, especially for small and medium enterprises, we provide
specialized client apps dedicated, for example, to inventory optimization or call center staffing optimization. Those apps take care of uploading your data toward Lokad and retrieving your forecasts. We provide built-in support for more than 30 common business apps such as Excel, QuickBooks, PayPal, Sage, ...
Cloud computing and scalability
Rule of thumbs forecasting methods are cheap to compute; a single personal computer being able to produce thousands of forecasts. Yet, if you want higher accuracy, then smarter statistical methods are needed. In our experience, advanced forecasting methods typically reduce forecast error by more than 20% compared to manual forecasts. But such smarter methods typically get you in trouble with traditional forecasting tools. Indeed, those methods happen to need thousands times more processing power and suddenly personal computers or regular company servers are not powerful enough anymore to deliver the forecasts in a timely manner.
Also, in our experience,
the number of forecasts required by companies grow fast as soon they start to realize how much profits they can make just by routinely adjusting their processes through automated forecasts. For example, a small retailer with 5000 product references and 10 points of sale will typically need more than 4x7x5000x10x30 = 40 millions forecasts per month (considering 1 month ahead daily forecasts) to fully automate their inventory replenishments.
40 millions forecasts per month for a medium company looks like an awfully high number. Yet, processing power has never been so cheap, and
1 month of processing time is now costing less than $10 (*) for a computer capable of roughly 2 billion elementary operations per second. Moreover, through large scale parallelization, that is to say, using many computers in the same time to speed-up computations, it is now possible to compress months of computations into hours.
(*) Amazon MapReduce pricing as of 2009-05-10, considering a single CPU at 2 GHz running without interruption for 1 month.Forecasting parallelization is a cornerstone of our technology. Since the initial launch of Lokad late 2006, we have been improving our analytics grid, that is to say a network of machines used to increase the data processing capacity of our technology.
Lokad has been the first forecasting provider to migrate toward cloud computing in 2009. We received extensive support from Microsoft teams to migrate toward the beta version of Windows Azure (now in production) – the cloud computing infrastructure of Microsoft. Windows Azure provides Lokad with the possibility to rent thousands of extra servers to accommodate large scale forecasting needs of our customers.
We believe that our technology is the most scalable forecasting technology available today, able to deliver unbounded forecasting capacities to our customers.
Modeling “real” customer demand through events
Sales, call volumes, cash flows can be represented as time-series, that is to say, lists of date/value pairs. Moreover, typical forecasting methods are relying on time-series analysis. Yet, one issue with time-series is that data do not always represent true customer demand. For example, when a store gets a product shortage, sales drop; yet, it doesn’t mean that actual customer demand is decreasing: it just reflects that there are fewer products to buy. In our experience, this sort of issue is frequent when events such as marketing campaigns or promotions impact the business.
When considering raw time-series data, statistical tools fail at getting a correct view of the business history, and consequently, fail at getting accurate forecasts. This is why Lokad has a richer data framework that also handles
tags and events in addition to
raw time-series. Through this framework, it becomes possible to “tell” Lokad about insightful historical events such as:
- A product promotion took place.
- A shortage was encountered.
- The call center had a power outage.
Those events are automatically analyzed to refine your forecasts to truly reflect future customer demand, instead of reflecting artifacts that lie in your historical data. Again, Lokad does not expect anything from its users but to actually provide relevant business data. In particular, users are never asked to figure out the actual impact of an event (such as a promotion), Lokad handles the analysis entirely on its own.
To our knowledge, Lokad is the one of the few forecasting providers that provides
built-in automated support for events. Instead, most competing technologies just expect users to perform complex data preprocessing operations on their own, manually cleaning up artifacts, or manually customizing statistical models with
external regressors to handle exogenous events.
Detecting similarities through “tags”
Trying to forecast sales of a lone product is a dreadful task. Indeed, considering that the average lifespan of a manufactured product on the market is less than 3 years, it means that, on average, a product has less than 18 months of sales history, that is to say less than 18 data points in case of monthly sales volumes. Obviously, 18 points aren’t much when it comes to statistical forecasting. In addition, if this
short history has also been impacted by marketing operations, then the forecasting task is complicated even further.
Traditional forecasting methods take a hard-hit when facing such problems. Typical tools ask one year of historical data or more – which precisely happens not to be available when forecasts are the most valuable, that is to say, at the beginning of the product lifetime. When not enough data are available, tools typically downgrade their methods to naïve approaches such as moving average that fail short at modeling seasonality or trend.
Lokad, on the other hand, uses a radically different approach. Forecasting sales of a lone product is hard indeed, but companies usually happen to sale many products, and frequently, many
related products. Thus, if the company is selling 10 related products, we have 10 times more data that can be exploited in order to refine the forecasts of the very product we are interested in. This approach is called concurrent time-series forecasting. Basically,
Lokad leverages correlations that exist between time-series to refine the forecasts produced for each time-series.
Yet, when time-series are too short, for example products with only a few months of history, there are simply not enough data to correlate time-series. This is why Lokad has introduced the notion of
tags. Tags are used to decorate time-series, for example telling Lokad about similarities that exist between products. By using tags, Lokad refines forecasts even when the sales history of a particular product is virtually nonexistent.
Again, as far we know, Lokad is the only forecasting provider that provides
built-in automated support for tags. Competitor’s products typically expect in-house experts to explicitly design the business rules that will be applied when the amount of data is not large enough.
Mixing Web2.0 and scientific computing
Structural risk minimization equation As we have seen, correlations between product sales, and more generally between time-series, is the key to refine forecasts when there are not enough data. Yet, the harsh reality is that there are never enough data. More data means lower forecast error, and lower forecast error, in turn, means higher profits. Thus, keeping up with the struggle to lower further forecast errors is one of the core values of Lokad.
Thus, we have decided to take the problem one step further: instead of restricting the input data to your sole company,
we are using all the data that we have at hand from the whole Lokad ecosystem to refine each forecast that we deliver. This also means that your data might be used to improve the overall accuracy of Lokad forecasts.
This process is completely secure. At no point, your data get actually transmitted to any 3rd party, and certainly not to other Lokad customers. Forecast’s refinement through multi-company data analysis is a very indirect process which can’t be reverse engineered to actually gain insights in your data.
Then, contrary to common intuition, the primary benefit of multi-company data analysis is not detecting correlations between companies. Correlations only come in second, but a long way behind
noise identification. Intuitively, when looking at a single midsize company, it is usually very hard to sort patterns, which represent repeatable events in the business, from market noise, that is to say plain randomness. Yet,
if you happen to have data from a few hundred companies at hand, sorting patterns from noise suddenly becomes a lot easier. For example, if the same seasonality happens to be found in another business, then the probability for this seasonality to be just a random artifact of the market becomes very low. Better noise identification means fewer mistakes while identifying business patterns, and, ultimately, better forecasts.
We believe that
our technology which combines social networking with statistical learning is pretty unique; and we are very excited at the idea that each new customer is directly improving forecasts for everyone.
Statistical models used at Lokad
The notion of
quality when considering a forecasting model is
very subtle. The key of statistical forecasting is that you need a model that
lowers the error, not against the data that you have, but
against the data that you don’t have, that is to say future data. We believe that this problem, commonly referred as
overfitting is too little known, while its business impact can be massive. Have a look at our video
Overfitting: when accuracy measure goes wrong for more details.
Lokad is relying on a
large custom library of statistical models that includes the well-known classics such as Box-Jenkins, exponential smoothing, autoregressive and all their variants. Yet, in our experience, the
classics typically poorly handle:
- Concurrent time-series correlations.
- Time-series tags (i.e. product descriptions).
- Very short time-series.
That’s why Lokad has been developing more complex models to handle those situations. We don’t pretend to have invented new mathematical theories (yet), as we are mostly relying on the
statistical learning theory.
Then, it must be noted that selecting the right statistical model is a task as difficult,
and sometimes more difficult than designing the model itself. Again, Lokad leverages large scale data correlations to refine its model selections. We are much more confident in choosing a model for a particular time-series if that model is also selected for many other related time-series.
Monitoring and ongoing improvements
Forecasts are delivered in a fully automated manner. Yet, this doesn’t mean that Lokad is a mindless forecasting provider, quite the opposite in fact.
We are continuously monitoring our forecasting systems. This process is done Lokad-wide, which enables us to significantly lower our internal costs because each expert at Lokad can monitor a large pool of companies at once.
Every day, we run forecast simulations
to carefully assess the weaknesses of our forecasting technology. This monitoring process is ongoing. You don’t have to do anything special but to upload data toward Lokad to benefit from it.
Through monitoring, problems are first identified, and then,
potential solutions become part of our development roadmap. We believe that we have just started to scratch the surface when it comes to statistical forecasting. Also, our Web2.0 approach makes it possible to explore solutions that were simply not even conceivable before.
Contrary to classical forecasting tools, choosing Lokad means that
your forecasts will be reviewed by experts, and that your forecasts will be compared to the ones obtained for roughly similar companies, vastly facilitating the process of assessing potential issues. Then, with Lokad,
your forecasts just naturally improve over time along with our technology. Choosing Lokad means that your company won’t be left behind the competition just because your forecasting models don’t get updated frequently enough.
Start forecasting now, $30 off for pre-paid forecasting services.