00:00:07 The role of analysts and consultants in assessing software.
00:00:28 How software companies are currently evaluated.
00:01:56 Complexity of supply chains and issues with the questionnaire approach.
00:03:43 Detailed critique of the questionnaire approach and comparison to restaurant reviews.
00:07:10 More focused approach to software assessment.
00:08:02 Common error of companies focusing on specific solutions instead of identifying their real problem.
00:09:00 Role of software vendors and the necessity of a concise summary of their product’s vision and design principles.
00:11:19 The prevalence of inconsistent features in enterprise software products and their poor real-world performance.
00:12:00 The history and evolution of analyst companies is discussed.
00:14:43 The shift in analyst companies’ business model to a pay-to-play system due to the widespread availability of information on the web.
00:16:02 Shift in business model of analyst companies post year 2000.
00:16:49 Consequences of this shift for the perceived neutrality and credibility of these companies.
00:17:42 Personal experiences and observations of the impact of this shift on the industry.
00:19:46 Analysis of how analyst companies have transformed into marketing agencies.
00:21:59 Speculation on the future of analyst companies and the evolution of their business model.
00:24:41 The current state of the software business and its relation to the operations of analyst companies.


In the interview, Kieran Chandler and Joannes Vermorel, founder of Lokad, discuss the inadequacies of current software evaluation methods, particularly in the supply chain field. Vermorel criticizes the common practice of using extensive checklists for assessments, arguing they oversimplify complex features and fail to capture nuances of software capabilities. He indicates that these questionnaires create an illusion of rationality and precision, but often miss the specificities of each software function. He also criticizes the pay-to-play model adopted by market analysts, arguing that it favors vendors who can afford to pay for better exposure, rather than objectively assessing software quality. Vermorel advocates for a more comprehensive and nuanced approach to software evaluation.

Extended Summary

In the interview, host Kieran Chandler discusses the complexities of software evaluation, particularly in the field of supply chain optimization, with Joannes Vermorel, the founder of Lokad.

Chandler begins by questioning the trustworthiness of software evaluations conducted by analysts and consultants. The discussion is motivated by the often subjective viewpoints of such individuals and aims to shed light on the assessment process. Vermorel, having experience in the field, shares his insights on the current methods employed to evaluate software companies.

Vermorel discloses that Lokad frequently receives requests to partake in various surveys, either from prospects or consultants. Most often, these assessments take the form of extensive questionnaires, with hundreds of checkboxes to fill out. These could be delivered via spreadsheets with dozens of tabs or online surveys. He expresses frustration at this practice, arguing that it’s not an effective way to evaluate enterprise software, particularly in complex areas like supply chain management.

Vermorel explains that the complexity of the real world, particularly in supply chain, reflects on the software designed to manage it. This complexity is represented through a multitude of features and capabilities, which in turn leads to lengthy questionnaires. However, he believes that these are more of a source of confusion than a tool for evaluation.

When Chandler probes further about the source of this confusion, Vermorel explains that most of these questionnaires are comprised of close-ended questions, which he feels inadequately capture the nuances of the software’s capabilities. He points out that many of these yes-or-no questions, such as compatibility with a certain language or support for a certain function, are misleadingly simplistic. For instance, a question about Excel support can be tricky, as the answer depends on what level and type of support is being referred to.

Vermorel stresses that these questionnaires create an illusion of precision and rationality while failing to capture the intricate details of the software’s capabilities. He uses the example of Excel exports, explaining that while Lokad does support this feature, there are specific nuances that aren’t covered by a simple yes-or-no question. For instance, Lokad may not support certain macros or options due to security concerns. He suggests that an entire chapter of a book could be written about the nuances of each binary question, illustrating the inadequacy of such questionnaires.

Vermorel characterizes the use of lengthy checklists as an unscientific and lazy way of giving the illusion of rationality. He illustrates this by comparing it to choosing a restaurant based solely on yes/no questions about the menu and cleanliness practices, without considering the overall experience or quality.

The use of these questionnaires, Vermorel argues, is common because they’re easily scalable and can be sold as a product, particularly by consultants advising companies on software choices. However, he criticizes this approach for its lack of nuance, noting that a large portion of the questions often don’t directly relate to the software or problem at hand.

Instead of relying on questionnaires, Vermorel suggests a more thorough examination of the problem that needs to be solved and the fundamentals of the product that might solve it. This examination should ideally fit on one page and describe the problem without already focusing on a particular solution. Vermorel emphasizes that companies often misidentify their needs; for example, they might request a forecasting software, but the real need is to improve supply chain performance.

Vermorel further criticizes the enterprise software space, where products often accumulate an array of inconsistent features over time. These products might be good at ticking off checkboxes but perform poorly in the real world due to their lack of vision and consistency.

The discussion then shifts to the origin and popularity of check-box questionnaires. Vermorel dates their emergence to the early 2000s, a period when the internet became mainstream, radically changing information accessibility. Prior to the internet, information was scarce, and businesses relied heavily on market analysts to identify potential suppliers. With the advent of the internet and search engines like Google, companies could easily find the information they needed, which Vermorel suggests should have made market analysts obsolete. However, the practice of using questionnaires has persisted.

Vermorel described how these companies used to sell reports to clients looking for suppliers or tech vendors before the year 2000. However, the advent of the internet and the subsequent ease of access to information led to a significant drop in the demand for these reports.

In response to this change, analyst companies pivoted their business model to a ‘pay-to-play’ approach, where tech vendors pay to be featured prominently in their reports. This shift marked a reversal from the previous model, where analysts maintained neutrality by not charging vendors to avoid conflicts of interest.

Vermorel explained that today, many analyst companies have grown larger than ever despite their value proposition being diminished. They’ve achieved this by charging software vendors to be showcased in their reports. He expressed skepticism over the fairness of this practice, noting that the more a vendor pays, the better they are portrayed in the reports.

The discussion then shifted to the evolution of these analyst companies. Vermorel described how they’ve transitioned from being neutral market analysts to essentially functioning as marketing agencies. The profitability of software vendors has fueled this transition, as they typically have more money to spend on promotional activities than regular companies would spend on reports.

Vermorel raised concern about the impact of this shift on the tech industry. He argued that these analyst companies serve the interests of tech vendors instead of providing unbiased advice to clients looking to acquire software products. This change in focus has led to a biased perspective that doesn’t genuinely assess the quality of the product but instead promotes those who pay more.

Vermorel concluded by expressing his hope that the market will catch up with the new situation, with vendors recognizing the limited value in paying for promotion. He also praised top-tier management consulting agencies for maintaining a clear separation between their business and what they recommend to their clients, ensuring they deliver value to the clients rather than focusing on charging vendors.

Full Transcript

Kieran Chandler: Today, we’re going to try and shed a little bit of light on the process and understand how much the supply chain practitioner can trust the results of their surveys. So, Joannes, I guess this is something you’ve got a bit of experience in. How is it that software companies are currently assessed at Lokad?

Joannes Vermorel: At Lokad, we receive, I would say, a couple of times a week, every single week, prospects or consultants who want us to take part in a survey of some kind. It can be a large company that directly wants to buy a new piece of software, in which case it’s going to be an RFP (request for proposal). Or it can be consultants or market analysts who just want to assess Lokad in general. Most of the time, and when I say most of the time, I mean like 99% of the time, it comes in the form of a long questionnaire with checkboxes. And when I say long, I mean several hundreds of checkboxes long. It usually takes the form of either a spreadsheet with dozens of tabs and checkboxes, or an online survey with the same thing on a webpage. And then you have to take hundreds of checkboxes.

Kieran Chandler: What is very infuriating to me is that it seems to be like the default practice to evaluate enterprise software. You end up with those exceedingly long questionnaires, which actually are more a source of confusion than anything else, in my opinion.

Joannes Vermorel: Okay, so let’s talk about that in a bit more detail. Where does the source of confusion tend to lie? I mean, these analysts and consultants have a fair amount of experience, so why do these questionnaires not work so well?

So, you have to think that the vast majority of questions are in close form. You end up with an enormous number of questions that are seemingly narrow. It’s a yes or no question, but in practice, there are tons of nuances in the answer. You end up with questions like, “Do you support Excel exports?” That would be a classical one. Most enterprise pieces of software need to be able to export to Excel, so that’s a question that is going to be asked. But this is actually a very complex question. Do you really want to support Excel export? And there are plenty of details. What is maddening is that those questions, with hundreds of checkboxes, give the illusion of rationality and precision, while actually most of those binary answers require a detailed explanation. For example, we can literally write a chapter of a book on our answer to say, “Yes, we kind of support Excel export. We maximize the capacity in terms of large Excel spreadsheets because it’s frequently useful for supply chain. But as far as macros and dangerous options such as loading from local files on your computer, then we don’t. So, actually, on purpose, we don’t exactly support the full specification of Excel spreadsheets.” But you see, that’s a lot of nuances. The same could be said for everything like a mock use, for example, which is more like a splash-in topic.

Kieran Chandler: But I guess the real benefit of taking this questionnaire approach is that you can send it out to thousands of different suppliers. I mean, there’s so much choice out there. Is there really a better way of doing it?

The lazy way, it’s not scientific. It just gives you an illusion of rationality. If we’re not discussing software, imagine reviewing restaurants. They’re all kind of the same, but if you were to check boxes like “do you serve meat?” or “do you serve French fries?” or “do you clean your dishes at 80 degrees after use?” and so on, would that really give you any insight into the restaurant? Would it help you decide if you want to go there?

Joannes Vermorel: I believe those questions are the lazy approach. Many parties in the market rely on them because they are easy to scale and become a product you can sell. Consultants can go to large companies and say, “We’ll help you pick the right software. Here’s a 600-check-box questionnaire that you send out to vendors.” Interestingly, regardless of the type of software, about two-thirds of the questions are completely independent of the actual problem. It’s maddening because I’ve seen supply chain software questions that are almost the same as marketing software questions. These questionnaires are just a lazy way to probe the market.

A better way is to assess the fundamentals of the product you’re trying to buy. What are the fundamentals of the problem you’re trying to solve? You should be able to summarize that in one page. When we try to explain what we do at Lokad, it takes hours. So, what characteristics would you want to fit into that one page? First, there are two sections.

The first page would describe the problem you want to solve. Companies often get this wrong. They start by saying they want a forecasting software. But forecasts are just a means to an end. What they really want is supply chain performance. So, instead of describing an intermediate problem that may or may not be part of the final solution, focus on the problem itself. Differentiate the problem from what you perceive as the correct solution. Vendors should propose solutions that make sense.

On the software vendor side, when assessing a vendor, produce a one-page summary that describes the vendor’s vision and core design principles. Don’t rely on checkboxes, but use words to understand what the vendor is really about.

Kieran Chandler: The first thing I wanted to ask you about is the product that software vendors are trying to push forward. It seems to be something complex, with hundreds of tech boxes. How do you see this?

Joannes Vermorel: Well, it’s not rational. If you were to describe a product as basic as Google or Facebook using these complex questions, you wouldn’t even recognize them. For example, asking if Facebook is compatible with C++, you might think it is because they have APIs, but Facebook is a complex platform with bits in C++ and many other parts in different languages. So, it’s unclear if it should be considered a C++ product. The problem is that many enterprise software products have a large number of inconsistent features, and although they are good at checking boxes, they are poor at actually doing anything. They lack a clear vision and consistency, and have just been accumulating features without purpose.

Kieran Chandler: So, you’re saying that these questionnaires and assessments used by analysts and consultants to evaluate software products are not capturing the right details?

Joannes Vermorel: Yes, exactly. They are somewhat lazy and fail to capture the essential aspects of the product. They are popular because they provide checkboxes and teachers, but they don’t truly make sense in the real world. These questionnaires have become popular, but they lack the ability to summarize a product in a meaningful way. If a product cannot be summarized in one page with a clear understanding of its purpose and approach, then it doesn’t make any sense.

Kieran Chandler: That’s interesting. So, where did these questionnaires come from and why did they become so popular?

Joannes Vermorel: It’s an interesting history. Until around the year 2000, information was scarce, and it was challenging to find suppliers or companies. For example, if you needed a supplier in Malaysia, it was difficult to know where to find one, especially if you couldn’t read the local language. Market analysts would provide extensive lists of companies and suppliers, but with the advent of the web and search engines like Google, finding information became much simpler. You could search for specific products or services and get a list of companies within hours. One might have thought that these search engines would replace the need for market analysts, but that’s not what happened. The questionnaires and assessments remained popular, even though they didn’t fully capture the capabilities and essence of the products.

Kieran Chandler: So, Joannes, I’d like to discuss the role of analyst companies in the software industry. Can you share your insights on this?

Joannes Vermorel: Certainly. So, prior to the year 2000, there were these analyst companies that would sell reports to companies. These reports contained information about various software and hardware vendors. Back then, people would buy these reports to find suppliers or software vendors. The analysts made money by selling these reports.

Kieran Chandler: Right. But then something changed?

Joannes Vermorel: Yes, with the advent of the internet, access to information became much easier. These analyst companies realized that their value proposition was no longer as relevant. So, they had to change their business model.

Kieran Chandler: How did they pivot?

Joannes Vermorel: They started targeting tech vendors, especially on the software side. They offered a “pay-to-play” model. Essentially, they said to vendors, “If you want to be prominently featured in our reports, you have to pay.” This was a complete reversal of their previous model. Before the year 2000, analysts would never charge vendors as it would create a conflict of interest. They aimed to remain neutral and unbiased. Instead, they sold their reports to client companies.

Kieran Chandler: That’s interesting. So, the analyst companies began charging vendors after the pivot?

Joannes Vermorel: Yes, exactly. Nowadays, these analyst companies are bigger than ever, but they generate revenue by charging software vendors. The more a vendor pays, the better their positioning in the reports. It’s a showcase of sorts. Officially, when you talk to these companies, they claim to never accept money from vendors. They deny any breach of trust or ethics. However, they do offer expensive programs to vendors for better market reach and positioning. In reality, it’s a way for them to charge vendors indirectly. I’m often approached by their salespeople on LinkedIn, offering these programs.

Kieran Chandler: So, they deny taking money from vendors but offer costly programs instead?

Joannes Vermorel: Yes, exactly. They say they never collect money directly from vendors, which would be unethical. However, they offer coaching programs that come with a hefty price tag. They claim they can help vendors improve their market presence. It just so happens that those who have acquired a lot of training from these companies end up classified as “visionaries” in their reports.

Kieran Chandler: I see. So, these companies have quadrants in their reports, and those labeled as “visionaries” are often the ones who have paid for extensive training?

Joannes Vermorel: Yes, that’s correct. The companies that invest a lot of money in these programs end up being classified as “visionaries” on their quadrants.

Kieran Chandler: Joannes, I’d like to start by discussing the role of analyst companies in the market. There seems to be a shift in their business model. Could you share your insights on this?

Joannes Vermorel: Absolutely. Over the past two decades, I’ve observed a significant change in analyst companies. Previously, they provided a neutral point of view to the market through reports that clients would purchase. However, things have shifted. These companies now charge tech vendors and other companies to promote their products. Essentially, they’ve transformed into marketing agencies. Interestingly, this pivot has made them much bigger and more profitable than ever before. Vendors, especially in the software industry, have more money to spend compared to regular companies who would purchase the original reports. So, the focus has shifted from being a market assessment entity to being a marketing agency. While there’s nothing wrong with being a marketing agency, what is not acceptable is pretending to be a neutral market analyst while being heavily influenced by vendors who are their primary clients.

Kieran Chandler: That’s an interesting perspective, Joannes. So, you’re saying that these companies have become biased due to their financial ties with vendors. How do you think the market will respond to this pay-to-play model? Will clients and the market realize the limited value being added?

Joannes Vermorel: The market, in general, has a way of developing antibodies to such situations. Companies, like the ones my parents worked for thirty years ago, no longer rely on these reports as much. The practice of purchasing expensive reports, once common, has significantly diminished. Nowadays, clients rarely buy these reports, and even when they do, it’s at a fraction of the previous prices. So, on the client side, this shift has already taken place. However, it’s maddening to see that on the vendor side, many competitors of Lokad still spend a significant amount of money on these analyst companies. They proudly display quadrants where they are classified as “visionaries” for having concepts as basic as safety stocks, service levels, and ABC analysis.

Kieran Chandler: You know, you would say I’m very partial and I’m just… But ABC analysis is official, yeah?

Joannes Vermorel: Yes, yes, yes. Maybe you can be a visionary having safety stocks which are completely unsafe. Maybe, I don’t think so. But back to the topic, it’s very funny because what I see right now is that the market has to catch up with the new situation. As far as real companies who really want information, I also see that very good consultants, let’s say tier one management consultant agencies, if you have to name three, that would be the BCG, McKinsey, and Collin Berry. Those sorts of folks, they kind of distance themselves from some vendors. They are very careful not to be promoting any specific vendors like a marketing agency. So, they have already done the right move, which is to have a clear separation between their business and what they recommend to their clients. Because their business is still to deliver value to the clients, not to basically charge vendors and take a cut. They will ultimately realize that it’s a waste of money. But that’s also very funny. I mean, software businesses have been highly profitable. There are tons of very, very profitable software companies out there. So, in a way, it’s just that there is not the same degree of urgency to just cut the budgets, even if it doesn’t serve… I would say it doesn’t help much when you’re very profitable. You can cut spending, but it’s not naturally your number one priority.

Kieran Chandler: Okay, we’ll have to wrap up there. But that’s everything for this week. Thanks very much for tuning in, and we’ll see you again in the next episode. Bye for now!