00:00:00 Forecast accuracy distracts, monopolizes management bandwidth
00:03:30 Time-series mindset ignores agency, pricing, substitution
00:07:00 Greater accuracy can reduce profits in practice
00:10:30 Accuracy blind to assortment gaps and thresholds
00:14:00 Granularity and habits matter beyond time-series
00:17:30 Improved metrics often fail real-world deployment
00:21:00 Forecasts create manual toil; automation remains blocked
00:24:30 Ignore censored demand? You lose basket economics
00:28:00 Stability beats accuracy for workable operations
00:31:30 Planning myths: fixed demand, normal lead times
00:35:00 Profit-correlated KPIs aren’t forecast ‘accuracy’
00:38:30 Economic KPIs outrank tidy statistical errors
00:42:00 Accuracy not directionally linked to profit
00:45:30 Focus on decisions; quantify economic drivers
00:49:00 Sample, iterate; target zero insanity decisions
00:52:30 Ditch S&OP; prioritize resource-allocation decisions
00:56:00 Unified KPIs with category-specific parameters
00:59:30 If blocked, switch companies to progress
01:03:00 Pitch change with P and L; concise proposals
Summary
Businesses chase forecast “accuracy” because it’s measurable, not because it makes money. The metric—MAPE, MAE—rewards reactivity and point estimates, ignoring agency, prices, cannibalization, lead-time variability, and basket effects. You can improve accuracy and worsen decisions: zeros “win” with intermittent demand; volatile forecasts whipsaw orders through ratcheted supply chains. FVA and S&OP add cost and delay. Start from decisions and economics: margins, stockout penalties, overstock risk, opportunity cost—modeled probabilistically—then iterate to “zero insanity.” Use unified P&L-grounded KPIs; if the firm won’t pivot, build the case or change firms.
Extended Summary
The fixation on “forecast accuracy” is a classic case of measuring what’s convenient rather than what matters. Companies pour managerial time and money into narrowing error metrics—MAPE, MAE, and the like—on the assumption that smaller errors mean larger profits. The evidence on the ground says otherwise: the correlation is often weak, sometimes negative. You can get “better” accuracy and worse decisions.
Why? Because time-series accuracy treats the future as an inert extension of the past. It assumes away agency, pricing moves, cannibalization, substitutions, and operational frictions such as lead-time variability and returns. It stares through a microscope while elephants—assortment design, customer behavior, cross-item basket effects—stomp through the room. In retail fashion, an accurate forecast that a misdesigned collection will fail is still a failure of management, not a triumph of math.
Pursuing accuracy often rewards instability. Make models hyper-reactive and you’ll score prettier errors while whipsawing purchase orders through ratchet-laden supply chains. With intermittent demand, forecasting zeros can “win” the metric and lose the business. At the tails—where economics actually live—accuracy around the average misses the costly realities: stockouts that pull entire baskets to a rival, overstocks that write off perishables, a $20 screw grounding an aircraft.
The institutional add-ons—FVA programs, meetings that ritualize “alignment” (S&OP)—layer cost on top of distraction. Slow, manual synchronization cycles are a tax on agility. Meanwhile, platoons of planners must “translate” point forecasts into workable decisions because the inputs are expressed at the wrong granularity and ignore the right uncertainties.
The alternative is not mystical; it is managerial. Start from decisions, not forecasts: what to buy, where to place, at what price, today. Attach monetary drivers to each decision—margins, stockout penalties, overstock risks, opportunity costs, staffing constraints—and treat uncertainty probabilistically. Iterate with practitioners via “experimental optimization” until the goal is met: not zero inaccuracy, but zero insanity. Keep a unified economic recipe across categories, tuning parameters where needed, because all categories compete for the same cash and capacity.
As for KPIs, if a mathematician would recognize your formula as “accuracy,” it probably isn’t economics. Real, profit-linked KPIs are messy, business-specific, and verbose—because reality is. And if a firm is structurally committed to the accuracy fetish, the rational choices are two: build a P&L-grounded case for change, crisply packaged for executives—or change companies. Incentives, not slogans, determine outcomes.
Full Transcript
Conor Doherty: This is Supply Chain Breakdown, and today we will be breaking down the hidden cost of forecast accuracy. You know who I am.
I’m Conor, Marketing Director here at Lokad. And to my left, the indefatigable Joannes Vermorel, Lokad founder and CEO. Now today, before we get started, comment down below: do you think that increased forecast accuracy is a KPI worth pursuing? If so, why?
Comment down below. You’re talking directly to me today. Anything you ask, any comments, any questions, I will ask directly to Joannes in about 20 minutes. And with—excuse me—and with that, Joannes, let’s get started. First question: what are we angry about today? What is our problem with forecast accuracy, the bedrock of business?
Joannes Vermorel: I wouldn’t go as far as being the bedrock of business—maybe the bedrock of supply chain management. Now, the real problem is that for almost all businesses, considering what they are doing right now, it’s a complete distraction. It’s a massive distraction that has been ongoing for decades.
When we are talking about the hidden cost, I would say the problem—the beef that I have with forecasting accuracy—is that, again, for almost all of the companies who operate supply chains, this thing, this artifact, is consuming more than half of the management bandwidth for very, very little upside. My point is that it is an enormous distraction, which can even be made worse when people start to instrument that in ways that just complicate the process around forecasting accuracy for, again, very, very little upside.
Conor Doherty: If you’re saying that the concept of forecast accuracy monopolizes bandwidth, are you limiting the criticism—or the cost—to just the abstract like time, focus, attention? Because many would argue, I’m sure, that there is also a concrete financial dimension to this as well.
Joannes Vermorel: Yes. First, when you start looking at the future—again, for almost all—when you start looking at the future, there are so many elephants that are completely ignored. Do you adopt the client perspective so that you can even see cannibalization and substitution? If you do not, I’m not too sure exactly what you’re doing with your forecast.
Do you take into account the pricing? Because whenever you want to project yourself into the future, you have to take into account your own prices and the prices of your competitors. You see, the peculiar problem that I have with forecasting accuracy is that the traditional supply chain mindset looks at the future exactly like an astronomer looks at the future position of planets. You have a complete symmetry between the past and the future—a time-reversal symmetry—and you assume that the future is just the extension of the past.
What is extremely strange is that, in the classical way to look at the future—hence forecasting accuracy—it’s as if the company had no agency whatsoever concerning the future. It’s just the time-series perspective. We have seen that cannibalization and substitution are problems. You have the pricing effect. You also have the fact that companies are going to spend almost all of the efforts on forecasting demand, but what about the other sources of uncertainty—lead times, returns, etc.?
Then we can go further and ask: what about the granularity at which you are forecasting? Even if we accept the premise that we are going to do those forecasts, which granularity are we talking about?
The time-series granularity—which assumes that you can look at the future in equally spaced buckets, per day, per week, per month—is not necessarily sensible. There are many businesses for which it’s not. And then you’re also ignoring uncertainty. That’s a problem with point forecasts.
It’s an enormous array. It’s like a herd of elephants—I’m not too sure what is the proper term in English for many elephants.
Conor Doherty: Either a herd or a stampede. I’m going to guess something like that. Stampede of elephants.
Joannes Vermorel: So many elephants in this room that are just being ignored, and many companies want to focus on accuracy. It really challenges that. For me it looks like you have an entire room where you don’t know anything about this room. You have a microscope; you’re looking at a tiny square millimeter, and you say, “OK, what I need is a bigger microscope.” I really challenge the idea that what you need is a bigger, better microscope.
Conor Doherty: Two key points here: one, it’s a herd of elephants; and two—just to at least be charitable here—you, in your very first opening salvo, talked about the almost totality of companies spending inordinate amounts of, let’s say, money and attention and resources—allocating resources—for very little upside.
Joannes Vermorel: Upside return, you mean? OK.
Conor Doherty: Before we deconstruct that too much, what is the steelman position for that? What do companies at least believe is the upside to what they’re doing?
Joannes Vermorel: They believe that, when they approach the future via time-series point forecasts and adopt a metric which is a single-dimensional metric like mean absolute percentage error (MAPE) or mean absolute error (MAE), or one of the typical metrics, they assume that minimization of this criterion is positively correlated with profits somehow. I really challenge this position.
Very frequently there is very little correlation or no correlation, and sometimes even inverse correlation, which means that a forecast that can be more inaccurate can actually make the company more profitable. That was, by the way, one of the first discoveries that I made at Lokad 15 years ago. We were delivering superior forecasts accuracy-wise, and very frequently it was actually creating chaos and hurting the businesses of those poor first clients. It’s counterintuitive, but it does happen.
One example: one of the easiest ways you can improve the accuracy of a forecasting process is to make the forecasting algorithm extremely reactive, meaning that when you have small variations in recent data, the forecasting algorithm will respond very strongly. The problem is that it means you have a forecast that is swinging up and down massively all the time. So it’s a more accurate forecast but also a very unstable forecast.
In a real-world supply chain, you have ratchet effects all over the place. Once you pass a purchase order to an overseas supplier, if the next day your demand forecast suddenly drops because it was just a fluctuation upward of your demand forecast, you’re stuck with the purchase order. This is typically a situation where increased accuracy does not translate into profitability; on the contrary, increased accuracy decreases your profitability.
Another thing—this was the situation that actually led us to probabilistic forecasts a decade and a half ago: when you have intermittent demand, if you forecast zero, it’s usually a very accurate forecast. Very frequently, if you have intermittent demand, you can have a forecasting model that will just forecast zero demand. From a statistical perspective, that’s going to be a very accurate forecast. But business-wise, if you forecast zero, you replenish zero, and you sell zero. You don’t make that many profits based on that.
That’s the kind of shenanigans you get when you chase accuracy. You have dozens of other problems. Again, the belief that you have a positive correlation between forecasting accuracy and profitability is misplaced. It’s really misplaced.
Conor Doherty: The term you said before was “it might seem counterintuitive,” and I think—having had this discussion with not only you but other people—that understanding the goal of what it is you’re trying to do can help with the degree of counterintuitiveness you experience.
For example, if you believe that higher forecast accuracy is the metric you’re optimizing for, then you saying a less accurate forecast is not necessarily more profitable sounds counterintuitive. If you’re talking about making better decisions and that’s the end goal you’re optimizing for, we’re not the only ones who have pointed out—Stephan Kolassa wrote a couple of years ago in Foresight—the idea of “decision insensitivity.” You might have a more accurate and a less accurate forecast, but they converge on the exact same decision because of, let’s say, an MOQ. Thus, the extra forecast accuracy didn’t buy you anything. The less accurate forecast was more profitable in terms of net income than the market.
Joannes Vermorel: You have so many problems with the time-series perspective. You end up chasing what is generally known as the McNamara fallacy. You have the wrong paradigm. You’re choosing the wrong metrics. The audience can look it up; there is a Wikipedia page on this—misguided generals who led the Vietnam War efforts from the U.S. side.
Let’s consider an example. You are a fashion company, a fashion brand, and your problem is that the collection you’re about to push to the market is entirely missing the one thing that would completely match the present trend. Let’s imagine that this is the problem. We have a collection that is about to arrive in the market, and it completely ignores the textile, style, pattern—whatever—that would be super trendy.
Conor Doherty: Bell bottoms in the ’70s.
Joannes Vermorel: And your forecast is very accurate. It predicts that your upcoming collection is going to perform miserably. Thank you. OK, you have a very accurate forecast. Thank you very much.
You see, a forecast will only forecast—again, we are talking of the classic paradigm, time series—it will only forecast the time series that you have. What about a product that you don’t have but should have? How will this be accounted for in your accuracy metric? The answer is: it isn’t.
The main problem is that, as a paradigm, it is very weak and ignores so many things that, for the almost totality of companies, it is a complete distraction because they have not done the sort of preliminary work to make it a relevant metric. If we were talking of a company that would be extremely mature, that can assess demand thinking in terms of abstract demand completely decoupled from the actual assortment that you have; that is already considering the many varieties of uncertainties—demand, lead times, prices, returns, etc.; that is already embracing a probabilistic vision; that is already embracing a functional vision—functional meaning that the future depends on decisions that have not yet been made—we’re back to sequential decision processes, etc.
If you’re taking a company that has done all of that, then maybe that’s the point where you can start asking yourself about accuracy metrics, which are going to be, by the way, relatively strange because we are most likely talking about probabilistic forecasts. But that’s a lot of foundations that need to be put in place first.
When I say “hidden cost of forecast accuracy,” it’s that it’s a complete distraction because those foundations are completely missing, and thus companies are chasing metrics that are extremely narrow-minded and do not reflect the long-term interests of their businesses.
Conor Doherty: To build on the retail example you gave—I remember you mentioned this a while ago about assortment optimization. You gave the example of: if you’re a fashion retailer, how many bright yellow t-shirts or bright pink t-shirts do you want in your collection? You’re not going to—well, it really depends why you have them. If you have them because you think you’re going to sell them, that might not be the case, because you’re not going to sell very many. But not having them makes your assortment look ugly, because then you’ll just have white shirts and black or blue jeans and the store will look quite drab.
So trying to capture the value of that pink or that yellow t-shirt in a time series is—“lossy” is the term you use. It’s a lossy representation of value, or something to that effect. I’m butchering it, I’m sure.
Joannes Vermorel: Yes, it is indeed a very lossy representation of the information about your store. And that’s where I say: what is the granularity? The problem with those forecasts is that they typically adopt a perspective that is not very smart.
If you want to think of a store, you would like to think: how could I actually optimize my inventory and my assortment decisions—assortment decision, inventory decision, pricing decision—to maximize the demand being served successfully from this store? That would be a perspective. Another angle would be: how do I maximize the long-term value of every single individual client?
This matters, for example, in fashion: if you give a discount to a client, that creates a bad habit. The client suddenly has an expectation to come back. When this client comes back, he or she will expect that there is a discount again. Again, we are looking at the future. We are implicitly making a forecast.
When people start discussing forecast accuracy, they are trying to articulate a judgment about the quality of their anticipation of the future. What I’m saying is that the sort of qualities you get in the quasi-totality of companies—locked into an extremely narrow-minded paradigm, which is point time-series forecasts—are so restrictive and so wrong that whatever numbers come up in terms of metrics from this paradigm are just a waste of time. It is a waste of time, and whatever improvement you think you’re going to get is just an illusion.
I have also seen anecdotal evidence—dozens and dozens over the years—data science teams who came up with a “20% more accurate” forecast according to MAPE or MAE or whatever, which never goes to production because it creates so many problems. Lokad, again one decade and a half ago, was part of those data-science-led companies facing these sorts of problems. It seems the problem is still ongoing, and recent developments like FVA—Forecast Value Added—are just making this even worse.
Conor Doherty: A private question has come through. You can send them privately if you don’t want to comment publicly; I’ll ask them at the end. But I will come back to the idea of FVA and accuracy—because that’s what I was just asked about. I’ll push on a little bit, because again the topic is the hidden cost of forecast accuracy.
Your book is on the table. Economics is a huge part of your overall way of seeing the world, and you repeatedly stress the importance of both direct and indirect costs, with opportunity costs being the indirect. So, in pretty concrete terms, what are the costs in practice of an enterprise focusing on chasing accuracy? How does that show up in day-to-day expenses or losses?
Joannes Vermorel: It shows up in your inventory managers or production managers—or whoever is making the actual decisions—spending a lot of time to twist the numbers so that they can finally end up with a semi-sensible decision. Companies sometimes wonder: it’s very strange, we have the forecast that is supposed to be accurate, and then we have simple rules to derive the decisions—what do we buy, what do we produce, where do we allocate the stock, what are the prices? It turns out that it takes an immense amount of manpower to convert those forecasts into actual decisions.
Why is that? Because the forecasts are so much nonsense that you need a lot of manpower, of thinking power, to do all the things that the forecast is not doing properly. Those people, those entire teams, are in fact doing all the work of properly thinking the future so that the decisions actually make sense.
That is one of the hidden costs: why is your supply chain not entirely automated in terms of decision-making processes end-to-end? The short answer is that your forecasts are extremely dysfunctional, and that is a much more severe problem than your forecasts being inaccurate.
They are dysfunctional in the sense that they are not even expressed in a way that lends itself to decision-making processes. It’s not the right granularity. It doesn’t focus on the right things. It does not have the nuance that is needed, etc. So you have an illusion of a quantified future, but that’s just an illusion. When we are talking about forecasting accuracy—again, from the classic perspective—you’re just chasing this illusion of quantifying the future.
Conor Doherty: I’m going to preface this by saying we’re not going to go into a diatribe on FVA, but just as an example of concrete costs: often, we know companies will spend a sizable amount of money directly and indirectly on software products to institute FVA. Again, we’re not having a comment on whether or not FVA works; it’s irrelevant. It’s listing it as a cost, an additional surcharge in pursuit of a metric, like: we want to see what’s adding, what’s decrementing or incrementing accuracy.
So it’s not just the attention; there are the salaries, correct? There’s the attention, there’s the opportunity cost, and in many cases— we know examples—there is also software intervention. You’re vendors, paying for this, etc. So there’s a lot.
Joannes Vermorel: Yes. And also, for example, if you add people in the loop, as is done in S&OP—part of the idea of S&OP is to somehow increase synchronization, increase accuracy. You want people to produce what the sales team is about to sell and what marketing is about to promote. You want company-wide synchronization. In a way, it is about chasing accuracy so that you have less de-synchronization between all those parts.
But this comes as a massive cost: this manual synchronization is extremely slow. For most companies it’s only every once per quarter, and some companies are doing it once a month, but that’s already very slow. Even if you’re among the very best companies practicing S&OP, that’s going to be a monthly cycle. In my books, this is extremely, extremely slow. That’s another cost associated with chasing this accuracy: suddenly, everything that you do lags by 30 days or more. This is not good.
Conor Doherty: I agree, and I just jotted down an addendum to that. Again, when we’re talking about the focus—KPIs, often forecast accuracy—you mentioned that it makes you effectively blind to many other sources of uncertainty because, for the most part, it’s demand. We’re forecasting demand, and we forecast demand in a very specific way, which is through the lens of the stuff for which you have time series.
Joannes Vermorel: Exactly. It’s not all the demand. It’s the stuff where—effectively what you’re doing almost exclusively is projecting sales according to your historical sales. We are very far from asking, for example: how do you even think of accuracy when, in the past, you had major stockouts, and so you have not observed the demand? There was a censorship because you only sold so much because you didn’t have more.
A very frequent example: what would the almost totality of vendors say—and consultants, and many textbooks? They would say, “Oh, just ignore the portion of history where you had those problems.” My answer is: hell no. If you run into a stockout—say you do a promotion in a hypermarket and you run into a stockout after two hours on Monday morning—you open, bam, two hours afterward you are out of stock. It is an information that is very significant. If the promotion is supposed to last ten days, running out of stock by the end of the ninth day is a completely different situation.
So the fact that you end up in a stockout doesn’t invalidate everything. You can still use this information, even if it comes with complications. Again, the problem I have with “accuracy” is that it is practiced with a defective paradigm almost everywhere. Thus, when you chase this accuracy as it is practiced by 99% of companies, it comes with immense cost and very little upside—if any.
Conor Doherty: To take your own example and peel the onion in terms of hidden costs: you gave the example of a stockout event. The traditional perspective would be, “You’re in a store; you don’t have eggs. Well, I’ve lost the sale of eggs.” What about the fact that most things will be bought in combination?
You forecast a certain level of demand for eggs; you had a stockout on eggs. “Well, I lost the value of those eggs.” What about the bread, the cheese, the milk, the ham, the fabric softener, the washing liquid? All of those things would likely have been in the basket. Eggs being a certain product—I know for myself, I’ll eat a lot of eggs. Look at me. If there aren’t eggs in a store, I go to a store that has eggs, and I take with me all the money I would have spent on all those other products.
Joannes Vermorel: That’s the typical case for probabilistic forecasting: most of the economic value in supply chain lies at the extremes. It’s a surprisingly high demand that creates a stockout, or surprisingly low demand that creates the overstock and, potentially, in case of perishables, an inventory write-off.
For almost all companies focusing on classic accuracy, this will be invisible because you’re focusing on the average or the median. This is also true in aviation: you have AOGs—aircraft on ground. You’re missing a screw worth $20, and bam, your A320 is grounded for a day because you’re missing a tiny part.
The idea that your accuracy—point forecasts—will reflect or correlate to the economics is just very incorrect in most situations. For this audience, what they have to remember is that very frequently it is literally negatively correlated. By improving your accuracy, you are making the situation worse. That happens very frequently.
Otherwise, the counterargument will be, “We are improving accuracy because, at least, it doesn’t hurt.” My answer is: oh yes, it can hurt. It very frequently does. That’s one of the key reasons why so few of those data science projects ever go to production: those “more accurate” forecasts create so many problems that they are vetoed. It creates immense frustration for data science teams: “But look, our forecast was 20% more accurate!” Practitioners are not backward; they intuitively grasp that those numbers are going to create immense problems for the company. They don’t necessarily have an end-to-end quantitative analysis to explain why, so the typical inventory planner looks at those forecasts—which are supposedly more accurate but create many not-so-visible problems—and says, “No, I don’t like it. I just want to keep my three-year flat average and stick with it.”
For the data science team, that seems insane—why this three-year average? It looks dumb. What they don’t see is that the sophisticated model with its time-series perspective is completely broken and is creating a lot of problems. For the inventory manager, this very naive three-year average has interesting properties: high stability, easy to understand, etc. Because there is so much extra work to capture the future correctly, at least this input doesn’t interfere with all the rest of the work that inventory managers, production managers, allocation managers, store managers, etc., need to do to take real-world decisions.
Conor Doherty: You literally just said “capturing the future,” and the substance has mostly been the direct and indirect expense associated with forecasting demand, particularly through time series. You mentioned earlier the importance of lead times, returns, etc. Why do you think the quasi-totality of companies are so gung-ho about forecasting demand and getting super, super accurate about that, but this other enormous, very common, very well-known source of uncertainty—we don’t talk about that?
Joannes Vermorel: That’s where I have to pitch in the book—chapter 7, “The Future.” What almost all companies are practicing is, in technical terms, a teleological vision, which assumes that you can, just like Gosplan for the USSR, project the demand one year ahead (or in the Gosplan, five years ahead), and then freeze it. Then the world game becomes a resource allocation problem. It becomes only about orchestrating the resources for that, and you assume that everything will be done reliably. If not, this is a problem you need to fix.
For example, lead time: the classic theory would say, “Don’t forecast the lead time. Just get suppliers that are reliable and will deliver on time.” Does this view survive in the real world? It doesn’t. Nevertheless, it is a perspective adopted in the quasi-totality of books. You have even more nonsensical visions: sometimes the authors will admit lead times can vary and say, “Let’s adopt a normal distribution,” which goes into positive probabilities for negative lead times—super strange when you think about it. You will find in textbooks authors who say, “Let’s have a normal distribution for the lead time,” which means it’s fine to have minus one day of lead time: you order now and you receive the product yesterday. It doesn’t make any sense. Nevertheless, it’s in software and the literature.
Conor Doherty: Joannes, thank you. I should point out my computer has been going haywire in the background. I thought there were no questions and then it just rebooted while you were talking, and there actually are many. I had no idea, and then suddenly I saw that there are quite a few. Give me one moment to process—Microsoft did the Windows update at the right moment, unbelievable. Just the right moment. Computer works perfectly fine, then we do a live event.
We’ll come back to a closing comment later. I will push straight to a comment. This is from Timur: “When forecasting KPIs are not correlated with profits, that means these particular forecasting KPIs have to be changed. We have a good experience with redefining KPIs to those where we see a correlation.” What do you think, Joannes?
Joannes Vermorel: Yes. But then we should really challenge: if you apply all the necessary changes so that your KPI becomes correlated with profit, can you still call it “forecasting accuracy”? Is it what is being called forecasting accuracy in the literature? Is it what is called forecasting accuracy within advanced planning systems or enterprise planning solutions? My answer is: no.
So, what if we say it’s possible to introduce a KPI that is actually correlated to profit? Absolutely. But it doesn’t go by the name of forecasting accuracy anymore. Nobody looking at this calculation would say, “Oh, it’s a variant of accuracy.” Yes, it’s possible to correlate the quality of your anticipation of the future to your profit. But when you do that, you end up with stuff so different that no statistician would call it accuracy anymore. We have left; we are so far gone that it’s something very different.
Conor Doherty: Next question from Vivek: “Should we measure accuracy or error in volume, or error in percent—accuracy in volume or error in volume?”
Joannes Vermorel: The problem is not to have an absolute metric expressed in units or something expressed as a percentage. This is a completely irrelevant technicality. Same thing if you want to pick the square root error or whatever. All of those are mathematical instruments with a mathematical definition. They have no correlation whatsoever to the profits of the company.
If you have a proper KPI, it will be something that—a litmus test—if a statistician or mathematician would recognize your performance indicator’s formula as something named accuracy, it’s not an economic criterion. When you factor the economics, it gets very dirty and extremely specific to the business. It becomes something you cannot transpose to another business. It becomes extremely attached to the strategic ambitions of this very business and comes with many edge cases. There is a lot of complexity specific to the business.
Those performance indicators that are economically driven are very useful. If you want to recognize them, they are typically quite verbose, because they need to take into account plenty of factors of the business. They don’t have the mathematical elegance of purely mathematical criteria like MAPE or MAE. This is not something you can write in one line of code. It will typically take hundreds, if not a few thousand, lines of code because you have to factor in tons of stuff.
Conor Doherty: I’m going to return to a question sent earlier. This was in response to your comments about a negative—or rather, a less accurate—forecast being more profitable. I’m reading this verbatim: “OK, 10% more accurate forecast might not make more money, but a 20% less accurate one will surely lose you money. Thus, is accuracy at least directionally correlated with profitability?”
Joannes Vermorel: No, it is not. That was the mistake I made in the first few years of Lokad. The business model of Lokad was: we deliver more accurate forecasts. And we did. We’re still quite good on forecasting accuracy. A few years ago, for the M5 forecasting competition on Walmart data, Lokad landed number one worldwide at the SKU level and number five overall, while none of our competitors—who were centering their discourse on “more accurate AI forecasts” and such—managed to be in the top 100.
Can a 20% less accurate forecast make your company more profitable? Yes, absolutely. This was the hard and painful lesson of the first few years at Lokad. How do you get a 20% more accurate forecast? By making the forecast more unstable. You have an algorithm that is very responsive to the latest drop of data. That is one of the easiest ways you can make your forecast more accurate. But when you put this sort of unstable, more accurate forecast into a real-world supply chain, it degrades performance. Going back to something 20% less accurate actually improves the situation.
One of the easiest ways to make a forecast “more accurate” with intermittent demand is to forecast zeros most of the time. Historically, we even won a major tender for a large European distributor by returning only zeros. We were forecasting demand for mini-markets—per product, per day, per mini-market—for five days ahead. The criterion was absolute value of forecast minus reality. I used my zero-forecaster model—return zero everywhere—and it did 20% better than the company number two in this forecasting competition for this tender.
Yes, you can improve the business by making the forecast less accurate. Time series are completely inadequate, and when time series are used, crazy things happen all the time. That’s why you need so many people tweaking numbers and nudging and working with spreadsheets on top of the forecasts—because you cannot translate directly time-series forecasts into decisions. That’s one of the core reasons why, since the late ’70s, supply chain automation didn’t happen: you cannot automate decision-making processes based on time-series analysis. That is the problem.
Conor Doherty: Next question from Dmitri—and thank you for helping me with the admin, Dmitri, and reposting it. Comment, then question: forecast accuracy is widely used because it’s easy. How do you, Joannes, describe your concepts to other business stakeholders—not necessarily the cool nerds who get it immediately?
Joannes Vermorel: The way we approach it: forget the forecast. This is a numerical artifact, a value that is only transient. It is a means to an end. What is the end? The decision: what do you buy, what do you produce, where do you put the inventory, at which price point?
Let’s have a look at the decisions. For every decision, let’s quantify—in euros or dollars—the half-dozen forces at play, and let’s have a debate on whether what we see through those forces feels right. For example: we put one unit in this store—what is the extra margin we think we’ll get by putting this one extra unit in this store today? Do we have a ballpark assessment? What are the stockout penalties we avoid—what is the improvement in quality of service? What is the risk of overstock we are creating? What is the opportunity cost of taking space in the store that could be used for a better product?
We need to have an agreement on those economic factors. Depending on the vertical, there will be plenty. The half-dozen or dozen economic forces at play will vary substantially from one company to another because the business model and strategic intent are different. Nevertheless, the method at Lokad is: express that into economic drivers, which reflect the future—future embedded into those economic valuations in monetary terms—then discuss whether we think we are in the right area with practitioners.
Very frequently we get higher-quality feedback when we take it from a purely financial angle. People will tell you, “Oh, you forgot: you’re telling me about pushing this unit today, but today the store is understaffed. They don’t have the resources to put the stuff on the shelf. If you still push something, someone at the point of sale needs to do things and clients will be poorly served. Thus, there should be a penalty.” OK, we include that as an extra factor.
So many things end up in this anticipation of the future. This is about getting it right for the future. That’s why I say “quality of anticipation” as opposed to “forecasting accuracy,” forecasting accuracy being time-series point forecasts.
Conor Doherty: Dmitri was listening, so there’s a follow-up: can you ask how to apply all these economic factors to a 5,000-SKU portfolio? You’ve dealt with larger than that.
Joannes Vermorel: Yes, considerably larger—we apply that to 50-million-plus SKUs. What’s important is writing the numerical recipe. The way you proceed with practitioners is sampling. You let yourself be driven by anecdotal evidence. Forget about having an average accuracy, average performance, etc.—this should not be driving you.
The method—also in the book—is called experimental optimization. You take an example, and the planner says “next, next,” they look at the SKU and say, “Ah no, this one I disagree.” If you start this method with actual people, they look at your recipe—not the code, the output, the economic factors—and they quickly object: “I disagree on this thing. For example, this product: you tell me the stockout penalty is this much, but this is diapers. For young parents it’s critical. If they don’t find the diapers of the correct brand in the hypermarket, they will immediately go to another hypermarket.” OK, so the penalty here is vastly underestimated.
They give you feedback. It’s anecdotal. Then it’s the work of the supply chain scientist to understand the general rule and get to the bottom of that. Our experience at Lokad—experimental optimization in practice—is: you do a first pass, planners object to 90% of your lines. No matter which SKU or decisions you pick, there are tons of objections. Then you iterate and iterate. Very frequently it takes us a few hundreds of iterations over two months—sometimes five iterations a day—tweak, repeat, tweak, repeat. Think of it as an Excel spreadsheet where you tweak things—an agile process.
Sometimes clients even have live discussions with the supply chain scientist over the phone. The scientist fixes the code during the call and runs it to see what you get. You iterate. At some point, the practitioner says, “I don’t have any objections anymore.” They look at the decisions: they look good and consistent; there is no more insanity. Our target for production is 0% insanity. We’re not looking at 0% inaccuracy; we’re looking at 0% insanity. That is a completely different perspective.
You do this through sampling. It’s pointless to say you want average performance because, when you average over many SKUs, you don’t see the problems. You don’t see the anecdotes, the special cases that need to be handled. Even if you look for profitability, you have many SKUs that are super profitable. If you average out, you can have a SKU where you do something poorly—insane—but it gets buried among many other SKUs that are profitable and sane. That’s why you need experimental optimization and an anecdotal perspective to fix the code rapidly.
Conor Doherty: Two more. I have to scroll back up—lots of comments. Dmitri, I hope that helped. OK, from—forgive me, I presume I’m pronouncing this correctly—Alif (or Leif): “From your perspective, what approaches can help organizations uncover and address hidden costs within cycles while ensuring a balance between agility and cost efficiency?” Parenthesis: cycles means S&OP.
Joannes Vermorel: Ditch S&OP. Those processes have only one upside: they make consultants rich. Just drop it. Really focus on the decision. Identify what decisions are being made. In the book I define a supply chain decision very simply: it is an allocation of resources that supports the flow of physical goods. That’s it.
You convert one dollar into raw materials for your supply chain—this is an allocation of resources. You take one unit of raw materials and convert that into a semi-finished product—allocation of resources. You move one unit of inventory from one place to another—allocation of resources. Focus on the allocation of resources; those are the decisions being made. Take everything from there.
Do not let numerical artifacts—intermediate steps—define your process. Forecasts are part of that; they are completely transient. They’re disposable. You can get rid of them, replace them by something better. They are not fundamental, unlike the decisions. The decisions are fundamental. Revisit your business 50 years from now—you will still have the problem of one dollar being converted to stuff you bought, transformed through a production process. Those decisions are extremely stable, unlike numerical artifacts, which are completely transient.
Conor Doherty: This is actually a long comment with a lot of context. I’m deliberately narrowing this down to just a question—we’ll send a longer answer later. Broadly: do you think KPIs should differ for different supply chain categories, mirroring differences in purchasing and production constraints and lead times?
Joannes Vermorel: Typically, no. You want KPIs that reflect the economics of your company. This can include many factors that are category-dependent. There are companies—some very large—with incredibly diverse businesses. If you have a company doing toys and aviation parts, that’s two separate businesses. Probably the KPIs are completely different.
But if we have something relatively homogeneous—say avionics—should you have KPIs defined differently depending on the type of avionics? Probably not. What you probably have is the code—the logic of your KPI—is the same across the board, but it has category-specific parameters. That’s typically the Lokad approach. Sometimes we even have product-specific or SKU-specific parameters. That’s fine. My suggestion is: try to keep the numerical recipe as unified as possible.
Why? Ultimately, everything you do competes for the same resources. All categories you purchase ultimately compete for the same dollars in the company’s bank account. Everything you keep in store ends up competing for the same storage space in the same warehouse. You have company-wide constraints. If things are not homogeneous, it’s extremely difficult to do proper arbitrage between allocations. That is also a problem with the classic view on supply chain optimization: they tend to process in silos, category by category. That completely misses the point. If you want to optimize your supply chain, it should be end-to-end, seeing where you can make an allocation of resources that has the highest rate of return in economic performance.
Conor Doherty: I should say there was an enormous amount of context to Timur’s question. If he was listening, he might be pulling his hair out like, “Conor has stripped my question down.” We’ll send a more detailed response later. Joannes has not seen the full context of that. That was an off-the-cuff comment.
We’ve been going for an hour. I think we’ve answered all the questions and comments, but there’s still one: a lot of people seem buzzed about this topic. Many operate in frameworks where they want to effect change but still have to respect their constraints. We deal in numerical recipes; we deal in constraints. They have S&OP meetings. They do have Forecast Value Added software that they have to, at least for now, operate with or around or through. What is your advice to people who want to start making the changes you’re talking about but are operating in that system?
Joannes Vermorel: Change company—literally.
Conor Doherty: Well, there you go, everybody. Thanks for having us.
Joannes Vermorel: I’m serious. The problem is that when you say, “OK, this framework, this organization is completely dysfunctional. There is something obvious that should be happening; it’s not happening,” you should be changing company. This will blow up. At some point a competitor will figure it out, enact the change, and for the company that does not enact the change, it will mean trouble.
Think of all those retailers who went bust facing Amazon. They literally could not comprehend what was happening. When I started Lokad, I had discussions in Europe with many retail companies—many have gone bankrupt since. They were telling me—because I was presenting Amazon as a threat—“Oh, Mr. ML, Amazon is just a niche, this tiny thing on the internet. It’s not serious. Nobody will ever buy”—insert here—“a TV, a sofa, a dress, a car, blah blah, on the internet. People love going outside.” They said, “They will never buy this or that online. Imagine buying an expensive camera online—no, obviously not,” etc. It completely blew up.
If you are in an organization where you have a lot of busy work, imagine your competitor decides to make a bold move and robotizes that. All of that is gone. How long will your company survive if they don’t do it? Do you think your position will still be there?
I see many people stuck. One of the privileges of modernity is that you’re not stuck in one place—especially for people with quantitative or analytical skills; those are in demand. Tons of companies are hiring. Lokad struggles to hire; it’s difficult. Why would you waste years of your life in a company misusing you through a broken process? This is nuts.
My suggestion: very politely, in a constructive way, try to push ideas to enact change. Frequently people are surprised: the reason change doesn’t happen is nobody is even trying to push for it. People assume by default the change would be rejected.
My limited experience is: you have a lot of leeway to go to higher-ups. If you have a case that makes sense, is well put together, and provides something reasonable and feasible, change can happen. That would be my suggestion.
But if you’re stuck in an obsolete process, it is urgent to move to another company that will make better use of you. Otherwise, imagine ten years from now: you’re still in the same obsolete position. On your resume you have ten years of busy work doing something obsolete. Selling yourself to your next employer will be very tough.
Conor Doherty: That’s not even theoretical. There are friends of the channel who’ve recently moved for that exact reason, and they’ve disclosed that—and good for them.
Joannes Vermorel: It’s also a way to create the change you want to see in the market. You see something obsolete and say, “I’m not going to contribute to that.” You go to a company doing something smarter—say Amazon, which happens to be very profitable—and the odds you will get a much higher salary after a while are very high.
Conor Doherty: As a closing thought, Timur agrees with you: “I agree with Joannes’s advice to change company. Do not waste your time if you’re not allowed to implement what makes sense to you.”
Joannes Vermorel: Yes, and if you want to have change, make a serious effort in packaging your proposal—that means really by pyramid style. You need a primer no longer than half a page for your boss’s boss, something very digestible. Then the longer version—maybe two pages—and then maybe ten pages, and a concrete example. If you can have numbers expressed in P&L—profit and loss—that will speak to upper management.
Don’t be the data scientist who says, “We need to go to deep learning, and I think we should really adopt low-rank decomposition; I think that’s the future.” Management will say, “What are you even talking about?” It needs to be very grounded. I don’t think I’ve ever met an executive who, when presented a business plan expressed in dollars or euros, would say, “I don’t even want to listen to that.” I have seen many situations where people say, “Your plan is interesting but completely flawed because you made the wrong assumption,” so what you compute is incorrect. But I’ve rarely seen top management unwilling to engage on something financially motivated.
Conor Doherty: Joannes, we have been talking and standing for 70 minutes. We’re out of questions. I did legs earlier today, so I am actually quite tired. We’re out of questions; we’re out of time. Thank you, as always, for all of your insights. And to everyone who’s attended and asked questions, both privately and publicly—much appreciated.
If you want to continue the conversation, feel free to connect with us privately, no problem. Or if you’re watching this video on replay—and it is available on replay—drop a comment down below and one of us will respond to you. And on that note, we’ll see you next week. And yeah, get back to work.
