00:00:04 Introduction and overview of supply chain forecasting problem.
00:01:28 Argument for forecasting error measured in dollars.
00:02:31 Preference for percentage error measurement examined.
00:04:15 Technological impact on improving forecasting accuracy.
00:05:49 Solutions for enhancing forecast accuracy discussed.
00:08:01 Weather forecasts’ role in probabilistic forecasts.
00:09:33 Issues with integrating weather data in supply chain.
00:11:29 Potential and challenges of human intelligence forecasting.
00:13:14 Optimizing expert knowledge to reduce forecasting errors.
00:14:24 Importance of data quality and improvement.
00:16:00 Forecasting difficulties in unpredictable industries like fashion.
00:17:33 Statistical forecasts in fashion and new product challenges.
00:18:01 Method proposal for new product forecasting.
00:19:02 Closing thoughts.
Joannes Vermorel, founder of Lokad, engages in a discussion with host Kieran Chandler about supply chain forecasting. Vermorel humorously unveils industry contradictions, revealing excessive claims of error reduction and advocating for a shift in focus towards ‘dollar errors’ rather than percentages. He observes that machine learning has revolutionized forecasting, with emphasis shifting towards insightful rather than merely accurate predictions. The pair also examine weather data but deem it too complex and unreliable for long-term forecasts. Vermorel points out the importance of human intelligence but suggests it’s impractical at scale, recommending an improvement in data quality instead. Despite the hurdles, Vermorel asserts Lokad’s model proves its effectiveness by accurately forecasting even for new product launches in volatile industries like fashion.
Kieran Chandler, the host, is engaging in a conversation with Joannes Vermorel, the founder of Lokad, about forecasting accuracy in the supply chain industry. There’s a noticeable discrepancy observed between practitioners who frequently complain about forecast accuracy, and software vendors who often claim very high forecast accuracies.
Vermorel identifies an industry anomaly, noting that for the past twenty years, at least one software vendor has been claiming a 50% annual reduction in forecasting error at every major supply chain trade show. Following this logic, Vermorel humorously suggests that we should theoretically have zero forecasting error now, contradicting the supply chain industry’s current state.
He continues by arguing that the obsession with percentage reduction of forecasting error is misleading, and might even be the wrong approach. Instead, Vermorel insists, the error’s impact is more significant when measured in terms of its monetary cost to companies. Businesses should be concentrating on ‘dollars of error,’ as businesses fundamentally exist to make profits, aligning better with this goal.
Addressing the industry’s continued focus on percentages, Vermorel characterizes it as a ‘mean absolute percentage error addiction’. He reasons that this preference is because percentages are more straightforward to understand and less likely to cause friction within a company as they don’t directly implicate a specific budget or person. Expressing errors in dollars would require someone to take responsibility for them, something that many risk-averse individuals in large organizations would rather avoid.
On the topic of technological progress and its influence on forecasting, Vermorel acknowledges that despite the aforementioned challenges, forecasting accuracy has indeed improved over time, largely due to advancements in statistical learning or machine learning. The application of these techniques to demand planning and forecasting has brought significant improvements to the supply chain industry.
Vermorel further clarifies that uncertainty is inherent when predicting the future, and therefore, it’s vital to assign probabilities to all possible outcomes. Interestingly, he proposes that forecasts don’t need to be more accurate but should provide more insights into the future. This advanced forecasting can assist in creating better supply chain decisions, the ultimate goal, and the basis for any physical, measurable impact. This reflects the dual aspects Vermorel’s firm focuses on - the forecast and the actual utilization of these forecasts in supply chain decision-making.
The conversation continues, delving into the influence of data on probabilistic forecasts. Vermorel concurs that forecasts are data-driven. Chandler raises a question about the incorporation of external factors like weather conditions in these forecasts as they often influence consumer behavior.
In response, Vermorel indicates that while weather data could be a source, it brings about two major concerns. Firstly, weather forecasts themselves are imperfect, and building forecasts on top of them could compound inaccuracies. Also, considering supply chain concerns, forecasts often need to be more than a week ahead, a period in which weather forecasts’ accuracy significantly drops.
Secondly, weather forecasts generate vast quantities of data that might be difficult to manage. Vermorel illustrates this by noting that one data point per hour per square kilometer every 20 minutes is generated when considering weather. This doesn’t only involve temperature, but also includes humidity, wind speed and direction, and light, leading to terabytes of data that are hard to incorporate into the supply chain.
The discussion shifts from the complexities of weather data to the potential of human intelligence in improving forecasts. Vermorel confirms the tremendous capability of the human brain but recognizes that it’s not a viable solution due to its high cost. For any large supply chain company, thousands to millions of supply chain decisions need to be made daily, requiring a considerable number of intelligent individuals. Most companies, however, can’t afford to employ that many people, which is why they depend on software companies. Vermorel accepts that while
human input can greatly enhance forecasting accuracy, it doesn’t scale, and this practical issue necessitates software solutions.
Vermorel stresses the need to use the intelligence of people within companies in a capitalistic sense. Their knowledge and insights should be treated as valuable resources that accumulate over time, rather than disposable or consumable entities. This approach can lead to continuous improvement in supply chain systems.
Vermorel suggests one way to make use of human intelligence is by improving the quality of data fed into the forecasting system. Data quality isn’t a given; it requires ongoing maintenance and enhancement efforts. Transactional data, Vermorel argues, plays a critical role in improving forecasting accuracy and has room for further improvements. For instance, few companies track stock-out history accurately. However, to predict future demand, it’s crucial to determine whether the lack of sales was due to no demand or a stock-out situation. Therefore, Vermorel champions for the proper recording of all relevant data, including stock-outs, promotions, and competitor pricing, to enhance forecast accuracy.
When addressing forecasting for products yet to be launched, Vermorel acknowledges the inherent challenge, especially in volatile industries like technology and fashion, where historical data is non-existent or erratic. He counters that the goal isn’t absolute forecast accuracy but rather a forecast that is more accurate than what teams can produce given their time constraints.
In situations like fashion, where new product launches lack historical data, Vermorel offers a solution. He suggests that while individual products might be new, they arise from a market that the company has observed through its own sales. Therefore, a statistical forecast can be built based on historical product launches and product attributes. This method can correlate a new shirt, for example, with previously launched shirts to build a forecast that, although it might not be extremely accurate due to the fashion industry’s volatile nature, can be accurate enough to be profitable.
Vermorel confirms that Lokad uses this approach in its forecasting model, allowing for effective predictions even under challenging conditions, such as new product launches.
Kieran Chandler: Today we’re going to be talking about a subject that’s divided a lot of opinion in the supply chain industry: forecasting accuracy. I’m delighted to say that today I’m joined by the CEO and founder of Lokad, Joannes Vermorel, who’s going to give me a bit of help with today’s discussion. So Joannes, thanks for joining us today.
Joannes Vermorel: Thank you, Kieran.
Kieran Chandler: Joannes, if you look at the supply chain industry as a whole, you often see practitioners complaining a lot about the accuracy of their forecasts. However, if you look at the software vendors, they often claim to have very high accuracies. They both can’t be right. So what’s your take on the situation?
Joannes Vermorel: The situation is indeed puzzling. Over the last two decades, maybe longer, at every single large supply chain trade show, there is at least one top-tier software vendor who claims to have reduced the forecasting error by 50% or something similar. Obviously, if you compound a 50% reduction of error per year over 20 years, logically, we should have no forecasting error at all, which is obviously not the state of the supply chain industry. Clearly, forecasting errors are still very present. Another way to look at it is that percentages are misleading. Thinking that you can reduce the forecasting error by X percent is the wrong way to look at the question. Forecasting errors cost money to companies expressed in dollars, not in percent. What we should truly be looking at are the dollar amounts of error for companies running supply chains in the real world.
Kieran Chandler: That makes a lot of sense because, fundamentally, businesses are there to make money. That’s basic economics, isn’t it? Companies want to maximize their profits and businesses profit from that. As a society, we can also profit because we’re getting a better range of products for lower prices. However, that’s not what we’re really seeing in the industry. The industry is still very hung up on using percentages. Why are they so hung up on that? What’s the reason for that?
Joannes Vermorel: The core of the MAPE (Mean Absolute Percentage Error) addiction is rooted in the fact that percentages are easy. It’s an easy thing to come up with a percentage. With a percentage, nobody gets really committed to anything and you’re not going to affect anyone’s budget. Expressing errors in dollars makes more sense, but it clarifies the stakes and who should be accountable for the error. Companies who operate large supply chains are made up of many people. These people, being part of a large organization, tend to be risk-averse. They don’t want to put forward something that would antagonize the rest of their own organization, and expressing the forecasting error in dollars does exactly that. It pinpoints the areas that are truly responsible for the dollars of error. So, while percentages are easy, dollars are the right metric. That’s probably why we’re still stuck with this situation; the easy thing is just easier to do.
Kieran Chandler: Okay, so if we put aside for now the dollars of error and just take the percentages of error as an example, is there any technological process that can be followed in order to improve the percentage of error? What can software companies do in that regard?
Joannes Vermorel: Despite the fact that forecasting errors have not improved by 50 percent a year for the last twenty years or so, the forecasting accuracy did improve. The bulk of the improvement was not driven by progress from within the supply chain world, but rather by a very broad technological progress from a domain known as statistical learning, more commonly known as machine learning. The latest flavor of that is actually deep learning. So, there has been real technological progress over the last 20 years.
Kieran Chandler: It’s known that using certain techniques for demand planning and forecasting in supply chain can lead to a significant measurable improvement in forecasts. It’s progress that can be seen in percentages, but also in dollars. So it’s very tangible. But there must be another way we can improve the accuracy of these forecasts. Could businesses and software vendors, for example, change the accuracy of their processes? Could they better train their teams, for example?
Joannes Vermorel: There are indeed two things that they can do. First, they can refine the scope of the forecasts themselves. What I mean is transitioning to probabilistic forecasts. The traditional type of forecast makes a single statement, like “my future demand will be exactly this.” Now, probabilistic forecasts take a much more holistic view. Many things can happen, there is irreducible uncertainty about the future, and so we assign probabilities to all those potential futures. That’s one way companies can improve, by adopting forecasts that tell more about the future. It doesn’t necessarily have to be more accurate, it just needs to provide more information about the future. That’s the essence of probabilistic forecasts.
The other angle is to think about what can be done to enable people to leverage these forecasts to make better decisions. Better forecasts are great, but can we turn them into better supply chain decisions? In the end, better supply chain decisions with a measurable impact are what truly matters. So those are the two angles we are looking at.
Kieran Chandler: And these probabilistic forecasts are driven by data, right?
Joannes Vermorel: Yes, absolutely.
Kieran Chandler: Regarding the data aspect, where do you draw the line? For example, things like the weather are quite interesting. In the summer, people are more likely to buy ice cream, while in the winter, they are more likely to buy hot chocolates, scarfs, and gloves. So, could we use things like weather forecasts in the probabilistic forecasts?
Joannes Vermorel: That’s a very broad question. To improve forecasting accuracy, you need to incorporate information, and this information has to come from somewhere. The first place you can look for this information is the historical data of the company itself. However, I believe that most companies, say 99% of them, are not fully exploiting the high-quality transactional data that they already have.
As for external sources, such as weather forecasts, they bring up at least two different concerns. First, weather forecasts are imperfect. If you want to build forecasts on top of other forecasts, you run into the problem of compounding forecasting inaccuracies. This is a challenging problem to solve in practice. Moreover, if you’re thinking of supply chain problems, you usually need to think more than seven days ahead, and the accuracy of weather forecasts becomes quite poor in practice. So using weather forecasts is problematic.
And then there’s another issue. For anyone who has tried to leverage these external data sources, they’ll realize there are challenges involved.
Kieran Chandler: So for weather forecasts, which are quite significant in statistical forecasting, we’re talking about extensive data. It’s not just one data point; it’s a data point every hour, per square kilometer, every 20 minutes looking ahead. We need to consider factors like temperature, humidity, wind, and light direction. So the data you would want to use to refine your supply chain forecast is immense. We’re literally talking about bringing terabytes of data. Can you discuss the practicalities of implementing such a vast amount of data?
Joannes Vermorel: Yes, indeed, the practicalities of incorporating something like global weather data into your supply chain can be staggering. It’s a challenging task. There are things much easier to do. So in practice, weather is probably not the best example to use.
Kieran Chandler: Okay, let’s consider another angle. What about the human brain? It’s an incredibly powerful tool. Is there a way we could harness it to improve our forecasts?
Joannes Vermorel: Absolutely. The algorithms we currently have are not fundamentally superhuman. They Excel in specific tasks, like playing Go or chess. But supply chain management is a very open problem that requires the full extent of human intelligence. A simple computer can’t outdo a supply chain specialist because it requires much more. However, the issue with human intelligence is not its capability but its cost. Large supply chain companies deal with thousands, if not millions, of supply chains daily. The question is how many smart individuals can you afford to make those necessary daily decisions? The answer, as seen with our clients, is usually not enough. So yes, human input can greatly improve forecast accuracy, but it doesn’t scale well, making it practically challenging. That’s why the industry relies on software companies like us.
Kieran Chandler: That’s an important point. How do you suggest companies make the best use of their smart employees? Should they apply their intuition and knowledge to the systems? How should this integration occur?
Joannes Vermorel: The key question here is how we can capitalize on these individuals to reduce the forecasting error. You don’t want to simply consume and discard their insights; that’s not the right approach. You need to capitalize on their knowledge over time for continuous improvement. One practical step they can take is improving the quality of data fed into the forecasting system. Maintaining and improving data quality requires ongoing effort. For instance, very few companies accurately track the history of stock-outs. But if you want to forecast future demand, you need to distinguish between no sales in a given period due to no demand, and no sales due to a stock-out.
Kieran Chandler: Are you properly recording all these? There are a lot of things like stock outs, promotions, your own prices, and the prices of your competitors that you can include in your data set. These are very actionable and can make your forecast more accurate.
Joannes Vermorel: Indeed, there are numerous factors to consider that can help improve forecast accuracy.
Kieran Chandler: Speaking of forecasting accuracy, how do you forecast for a product that has never been launched yet? In industries such as technology and fashion, there’s no historical data and these industries are quite erratic. Is there any hope of getting accurate forecasts?
Joannes Vermorel: That’s a very tricky question. It’s not about getting an accurate forecast in an absolute sense. If it was about predicting the next trendy product within the fashion marketplace for the next year, I wouldn’t be doing a statistical forecast. I would be playing the stock market. The real question is how to produce a forecast that’s more accurate than what your team can produce given their limited time. It’s not about absolute accuracy, but more about relative accuracy.
Kieran Chandler: So, you’re suggesting that there’s no hope for an absolutely accurate forecast especially in fashion?
Joannes Vermorel: In an absolute sense, no, fashion is too erratic. However, we can strive for something that is comparably more accurate. The challenge is, can we have a statistical forecast that works at all in fashion, where the products you want to forecast have no historical data? It’s puzzling because statistical forecasting relies on data. But here’s an angle. If you’re a fashion company, you are launching thousands of products every year. Even if it seems like a completely new product, it’s not entirely new. It emerges within a market that you can observe from your own sales. So if you want to build a statistical forecast, you need to leverage all your past historical launches and the attributes of the product.
Kieran Chandler: So, you are saying that we can still use past data to make accurate predictions?
Joannes Vermorel: Exactly. You can correlate this new shirt you’re launching now with shirts that you’ve launched in the past. What remains is the complete erratic nature of the fashion industry, which will be reflected in the forecasting inaccuracy. The point is that yes, you can tackle this, and you can produce a forecast that is accurate enough to be profitable even for product launches.
Kieran Chandler: That sounds like what you’re doing at Lokad?
Joannes Vermorel: Indeed, that’s exactly what we’re doing at Lokad.
Kieran Chandler: Thank you for your time and for this discussion. We hope our listeners found it enjoyable. If anyone has questions, feel free to get in touch, drop us an email, or leave a comment below. We may be able to discuss some of the more interesting questions in the coming weeks. Until then, thank you very much for joining us today and we’ll see you again very soon.