Discuss (0)

Print

RSS

Forecasting Technology

Ever since the 19th century, retailers have been routinely relying on sales forecasts to optimize their inventory levels. Today, Lokad is bringing something new to the forecasting field.

Our approach is simpler, more cost effective and also more accurate than traditional methods. This page aims at giving you an overview of what makes it all possible: our technology.

Check the Forecasting Technology FAQ for specific questions about accuracy, trends, seasonality, promotions, product launches, product lifecyles, cannabalization and more.

Abstract

Given the countless weaknesses and costs inherent to in-house forecasting, Lokad provides Forecasting as a Service, with quantile forecasts that represent a breakthrough for inventory optimization. It is designed for 3rd party integration, so that the data is fetched automatically, and the results are seamlessly integrated into your existing systems. We provide a new approach to forecasting that dramatically improves forecast accuracy by relying on Cloud Computing and scalability. We don't make the common mistakes of forecasting software and we model “real” customer demand, taking special events into account. We further improve accuracy by detecting similarities between products or series through “tags”. We even take the idea further by leveraging all our data for every single customer, combining Web2.0 and scientific computing. We use complex statistical models, and we are constantly monitoring and improving the quality of your forecasts.

The LOKAD forecasting engine

Forecasting as a Service

In-house forecasting solutions are often very costly and end up proving disappointing. They require hiring at least one qualified statistician, designing a clever model for your business, and purchasing expensive software licenses and servers to run the computations. These costs are so high that usually, only very big companies can afford them. And the results remain uncertain, to say the least.

Lokad works in a different way: we provide forecasting as a service. Lokad fetches your data from your system, and sends forecasts back. As soon as the data are retrieved, your forecasts are computed automatically. This architecture guarantees that your forecasts will be delivered in a timely manner even if your schedule is tight.

Our technology is designed to deal with your data in their current form. Traditional forecasting tools require you to prepare your data very carefully, which can prove extremely technical and time consuming: for example, you need to get rid of exceptional points, and later on, you are asked to provide expert insights about your data such as the presence or absence of seasonality. In other words, only after you've done all the work will traditional methods start giving results.

Lokad doesn't work like that. We deal with all these steps on our side. While careful preparation before sending data to Lokad, might improve the accuracy of your forecasts, if you don't have the time or the expertise, we can accommodate your data as is, and still deliver the best forecasts.

Quantile Forecasting

As of March 2012, Lokad has become the first software vendor to offer an industrial-grade quantile forecasting technology. Quantiles represents a major breakthrough as far inventory optimization for retail, wholesale and manufacturing businesses is concerned. Against our own classic forecasting technology (available online since December 2006), our benchmarks indicates that switching to quantile forecasts bring on average either 20% less inventory or 20% less stockouts, depending how the extra accuracy is used. However, since Lokad is already outperforming accuracy-wise traditional forecasting software, total benefits brought by Lokad are typically higher.

Designed for 3rd party integration

Getting forecasts is one thing. But what can you do with them once you have them? If you are a retailer, you probably want to derive your optimal reorder points. If you are a call center, you probably want to know your optimal staffing. Forecasts alone are useless: you need to be able to use them. In other words, in order to benefit from Lokad, companies need to integrate our forecasts right into their business. Lokad was designed, right from the start, to facilitate 3rd party integration.

We expose a Forecasting API (Application Programming Interface - a machine-to-machine communication protocol) that relies on an ubiquitous approach named REST. This approach is supported by all major software companies such as Microsoft, IBM, SAP, Google, etc. The Forecasting API enables programmatic integration of Lokad into virtually any 3rd party software as long as an Internet connection is available.

Breadth vs Depth: Changing the approach to forecasting

It's a very simple idea, and yet very few do it: instead of looking at a single product's history (product A) to forecast its future evolution, we use other products as well (products B, C, D...) and use their similarities in order to better forecast the product's evolution. So, to forecast product A, we don't just use A. We also use what products B, C, D, ... can teach us about product A.

A consumer product has, on average, a 3 year lifecycle. This means that, on average, the amount of data available for every single product is about 18 months. From a statistician's point of view, if we're only using A to predict A, we only have 18 data points to work with.

With just 18 data points alone no matter how smart or advanced your forecasting theory is, you really don't have enough data to work with. With 18 points, even a pattern as obvious as seasonality becomes a challenge to observe because you don't even have two complete seasonal observations.

Your mileage may vary from one industry to the next, but unless your products stay in the market for decades, you are most likely to face this issue.

As a direct consequence, classical forecasting toolkits require statisticians to tweak forecasting models for each product because no non-trivial statistical model can be robustly fit with only 18 points as input data.

With Lokad, you don't need a statistician. All the magic lies in the 90 degree rotation: our models do not iterate over data a single time-series at a time, but against all time-series at once. Thus, we have a lot more input data available, and consequently we can succeed with rather advanced models. This approach is just common sense: if you want to forecast the seasonality of your new chocolate bar, the seasonality of other chocolate bars seems like a good candidate. Why should you treat each chocolate bar in strict isolation from the others?

Simple? Well, not quite: from a computational perspective, this makes the problem a lot harder. The computing power that is required becomes many orders of magnitude higher. For example, if you have 10,000 SKUs the number of associations between two SKUs is roughly 100 millions (and 10,000 SKU is by no means a large number).

That's precisely where the cloud kicks in: even if your algorithms are well-designed to avoid a strict quadratic complexity, you're still going to need a lot of processing power. The cloud just happens to make this processing power available, on demand, and at a very low price.

Cloud computing and scalability

As previously mentioned, rough "rule of thumb" forecasting methods require little computing power; a single personal computer is able to produce thousands of forecasts.

Yet, if you want higher accuracy, smarter statistical methods are needed. In our experience, advanced forecasting methods typically reduce forecast error by more than 20% as compared to manual forecasts. But smarter methods typically get you in trouble with traditional forecasting tools.

Furthermore, these methods require thousands of times more processing power, making "in-house" solutions out of the question. This is why Lokad relies on cloud computing: in order to leverage all the computing power needed to give you the best forecasts.

In our experience, the number of forecasts required by companies grows quickly as soon they start to realize how much profit they can make just by routinely adjusting their processes through automated forecasts. For example, a small retailer with 5000 product references and 10 points of sale will typically need more than 40 million forecasts per month (base on 1 month in advance daily forecasts) to fully automate their inventory replenishment.

40 millions forecasts per month for a medium-sized company may seem like an awfully high number, but with Lokad, it is affordable.

Moreover, we use large-scale parallelization, accessing many computers at the same time to speed up computations. It is now possible to compress months of computation into hours.

Forecasting parallelization is a cornerstone of our technology. Since the initial launch of Lokad in late 2006, we have been improving our analytics grid, which is a network of machines used to increase the data processing capacity of our technology.



Lokad was the first forecasting provider to migrate toward cloud computing in 2009. We received extensive support from Microsoft teams to migrate toward the beta version of Windows Azure (now in production) – the cloud computing infrastructure of Microsoft. Windows Azure provides Lokad with the possibility of renting thousands of extra servers to accommodate the large-scale forecasting needs of our customers.

This project was a success and resulted in Microsoft picking us out of 3000 applicants as their Windows Azure Partner of 2010.

We believe that our technology is the most scalable forecasting technology available today, with the ability to deliver unbounded forecasting capacity to our customers.

Modeling “real” customer demand: taking special events into account

Sales, call volumes and cash flows can be represented as time-series, that is to say, as lists of date/value pairs. Moreover, typical forecasting methods rely on time-series analysis. Yet, one issue with time-series is that the data do not always represent true customer demand. For example, when a store runs short of a product, sales drop; yet, it doesn’t mean that actual customer demand is decreasing: it just reflects that there are fewer products to buy. Mistaking this for a decrease in demand hurts your business, and some forecasting solutions that make that mistake will still claim 100% accuracy: they forecast 10 sales and you sell 10 items, even though you could have sold a hundred. In our experience, this sort of issue is frequent when events such as marketing campaigns or promotions impact business.

When considering raw time-series data, statistical tools fail at obtaining a correct view of the business history, and consequently, they fail to generate accurate forecasts. This is why Lokad has a richer data framework that also handles tags and events, in addition to raw time-series. Using this framework, it becomes possible to “tell” Lokad about meaningful historical events such as:

• A product promotion took place.
• A shortage was encountered.
• The call center had a power outage.
  

These events are automatically analyzed to refine your forecasts to truly reflect future customer demand, instead of reflecting artifacts that lie in your historical data. Again, Lokad does not expect anything from its users other than the relevant business data. In particular, users are never asked to figure out the actual impact of an event (such as a promotion) on sales, Lokad handles the analysis entirely on its own.

To our knowledge, Lokad is the one of the few forecasting providers that provides built-in automated support for events. Instead, most competing technologies just expect users to perform complex data preprocessing operations on their own, manually cleaning up artifacts, or manually customizing statistical models with external regressors to handle exogenous events.

Detecting similarities through “tags”

Trying to forecast sales of a single product is a daunting task. Given that the average lifespan of a manufactured product on the market is less than 3 years, on average, a product has less than 18 months of sales history, that is to say less than 18 data points if sales volumes are reported monthly. Obviously, 18 points aren’t much when it comes to statistical forecasting. Moreover, if this short history was also impacted by marketing operations, then the forecasting task is complicated even further.

Traditional forecasting methods take a hard-hit when facing such problems. Typical tools ask for one year of historical data or more – which is never available at precisely the moment when forecasts are most valuable, at the beginning of the product lifecycle. When insufficient data are available, tools typically downgrade their methods to naïve approaches such as moving average, which fall short at modeling seasonality or trends.

Lokad, on the other hand, uses a radically different approach. Forecasting sales of a single product is indeed difficult, but companies usually sell many products, and often many related products. So, if the company is selling 10 related products, we have 10 times more data that can be exploited to refine the forecasts of the specific product we are interested in. This approach is called concurrent time-series forecasting. Basically, Lokad leverages correlations that exist between time-series to refine the forecasts produced for each time-series.

Yet, when time-series are too short, for example products with just a few months of history, there are simply not enough data to correlate time-series. This is why Lokad has introduced the notion of tags. Tags are used to decorate time-series, for example telling Lokad about similarities that exist between products. By using tags, Lokad refines forecasts even when the sales history of a particular product is virtually nonexistent.

Again, as far we know, Lokad is the only forecasting provider that provides built-in automated support for tags. Competitor’s products typically expect in-house experts to explicitly design the business rules that will be applied when the amount of data is not large enough.

Mixing Web2.0 and scientific computing: every additional customer provides additional accuracy for everyone

As explained just above, correlation between product sales and, more generally, between time-series, is the key to refining forecasts when there are not enough data. Yet, the reality is that there are never enough data. More data means lower forecasting error, and lower forecasting error, in turn, means higher profits. This is why we are always striving to lower forecast errors.

Accordingly, we're taking our "90-degrees" approach one step further: instead of restricting the input data to just your company, we are using all the data that we have at hand from the entire Lokad ecosystem to refine each forecast that we deliver. This also means that your data might be used to improve the overall accuracy of Lokad forecasts.

This process is completely secure. At no point does your data get transmitted to any 3rd party, and it is certainly not shared with other Lokad customers. Forecast refinement through multi-company data analysis is a very indirect process which can’t be reverse engineered to gain actual insight into your data.

Contrary to common intuition, the primary benefit of multi-company data analysis is not detecting correlations between companies. Correlations are of secondary importance, far behind noise identification. Intuitively, when looking at a single midsize company, it is usually very hard to distinguish patterns, representing repeatable events in the business, from market noise, i.e. plain randomness. However, if you happen to have data from a few hundred companies at hand, separating patterns from noise suddenly becomes a lot easier. For example, if the same seasonality happens to be found in another business, then the probability for this seasonality to be just a random artifact of the market becomes very low. Better noise identification means fewer mistakes when identifying business patterns, and, ultimately, better forecasts.

We believe that our technology, which combines social networking with statistical learning, is pretty unique; and we are very excited at the idea that each new customer is directly improving forecasts for everyone.

Statistical models used at Lokad

The notion of quality when considering a forecasting model is very subtle. The key to statistical forecasting is that you need a model that lowers the error, not against the data that you have, but against the data that you don’t have, that is to say future data. We believe that this problem, commonly referred to as overfitting is not well known, even though its business impact can be massive. Have a look at our video Overfitting: when accuracy measure goes wrong for more details.

Lokad relies on a large custom library of statistical models that includes the well-known classics such as Box-Jenkins, exponential smoothing, autoregressive and all their variants. Yet, in our experience, the classics typically poorly handle:

• Concurrent time-series correlations.
• Time-series tags (i.e. product descriptions).
• Very short time-series.
  

That’s why Lokad has been developing more complex models to handle those situations. We don’t claim to have invented new mathematical theories (yet), as we are mostly relying on the statistical learning theory.

Then, it should be noted that selecting the right statistical model is as difficult as designing the model itself, and in fact is sometimes more difficult. Again, Lokad leverages large scale data correlations to refine its model selection. We are much more confident in choosing a model for a particular time-series if that model is also selected for many other related time-series.

Lokad is constantly monitoring and improving the quality of your forecasts

Your forecasts are delivered in a fully automated manner. This doesn’t mean that Lokad is a mindless forecasting provider - quite the opposite in fact. We are continuously monitoring our forecasting systems. This process is done Lokad-wide, which enables us to significantly lower our internal costs because each expert at Lokad can monitor a large pool of companies at once.

Every day, we run forecast simulations to assess carefully the potential weaknesses of our forecasting technology. This monitoring process is ongoing. You don’t have to do anything special to benefit from it other than uploading data to Lokad.

Monitoring allows us to first identify problems and then incorporate potential solutions into our development roadmap. We believe that we have just started to scratch the surface when it comes to statistical forecasting. Our Web 2.0 approach makes it possible to explore solutions that were simply inconceivable before.

In contrast to classical forecasting tools, choosing Lokad means that your forecasts will be reviewed by experts, and that your forecasts will be compared to ones obtained for roughly similar companies, thereby vastly facilitating the process of assessing potential issues. With Lokad, your forecasts just naturally improve over time along with our technology. Choosing Lokad means that your company won’t be left behind by the competition because your forecasting models don’t get updated frequently enough. With Lokad, you're always in the race.

Start forecasting now, $30 off for pre-paid forecasting services.

Videos

FAQ

• What forecasting models are you using?
• How accurate are your forecasts?
• What about forecasting competitions?
• Do you evaluate the accuracy of your forecasts?
• How much historical data do you need?
• Macro trends (ex: financial crisis), how are they handled?
• Seasonality, trend, how are they handled?
• Easter, Ramadan, Mother's day and other quasi-seasonal events?
• Promotions, how are they handled?
• Product Life Cycles and product launches?
• Intermittent / low volume products?
• Cannibalization, how are they handled?
• Weather, how is it handled?
• Lost sales caused by stock-outs, how are they handled?
• Exceptional sales, how are they handled?
• Aggregation, top-down or bottom-up?