00:00:00 The decline of traditional consultants
00:02:33 Issues with vendor-driven infomercials
00:05:05 Problems with quadrant demarcations
00:07:11 Reports not challenging vendor claims
00:09:30 Importance of technical merits
00:11:11 AI vendors enhancing deep research
00:13:22 AI agents organize web search
00:15:36 Improving LLMs with instructions
00:17:27 Exaggerated claims benefit vendors and clients
00:19:55 LLM report ranking based on substantiation
00:21:56 Rapid detailed report generation
00:23:21 LLMs outpace consultants in tech knowledge
00:26:03 Precision in research using LLMs
00:28:09 Vendors providing neutral market views
00:30:46 Garbage data challenges in LLMs
00:33:51 Advantages of LLMs in software research
00:36:19 Biasing prompts undermines LLM objectivity
00:39:35 Learning prompting skills quickly
00:42:15 Reduction in misinformation risk
00:45:21 Efficiency in rapid market studies
00:47:37 Consultants as corporate companions
00:50:40 Consultant competency over bias
00:52:07 AI training boosts task performance
00:54:51 Executives prefer clear, well-written analysis
00:57:27 AI cuts time, costs in research
00:57:55 Interview concludes with gratitude
Summary
In a conversation on LokadTV, Conor Doherty and Joannes Vermorel discuss market research in supply chain management, critiquing traditional methodologies and exploring the transformative role of AI and Large Language Models (LLMs). Vermorel challenges the biases within conventional market research, driven by vendor interests, and extols AI’s capability to provide comprehensive, unbiased reports. The dialogue emphasizes AI’s promise in swiftly delivering detailed insights compared to traditional approaches, despite concerns about data quality and technological bias. Vermorel and Doherty reflect on the potential obsolescence of traditional consulting firms, advocating for innovative collaborations between AI and human expertise to redefine supply chain intelligence.
Extended Summary
In a recent dialogue hosted on LokadTV, Conor Doherty, Director of Communication at Lokad, engages with Joannes Vermorel, Lokad’s CEO and Founder, to unpack the complexities surrounding market research in the realm of supply chain management. This engaging conversation centers around traditional market research methodologies, the transformative role of AI, specifically Large Language Models (LLMs), and the evolving landscape for software vendors and consultants.
Doherty opens the discourse by addressing the enduring dilemmas companies face in selecting software vendors, noting the potential pitfalls instigated by conflicts of interest with software providers. Vermorel critiques the conventional approaches to market research, highlighting how financial incentives often skew reports in favor of enterprise software vendors. Such reports are typically riddled with deficiencies due to analysts’ lack of deep expertise, culminating in analyses that are neither insightful nor fulfilling.
Doherty seeks to understand whether Vermorel’s critique resonates broadly within the industry. Vermorel underscores that major market research reports frequently fail to provide meaningful insights into competitive landscapes, characterized by poor analyses that serve merely to propagate vendor-biased narratives. This critique extends to popular market assessment tools, such as quadrants, which Vermorel argues are shallow and lack visionary insights, except in cases like Brightwork.
Further discussing Brightwork, Vermorel attributes its distinguished approach to employing actual software engineers who independently assess intricate enterprise solutions, maintaining immunity to vendor biases. Shifted focus then lands on AI’s potential to radically alter market research, leading Doherty to query how AI, especially LLMs, addresses existing challenges. According to Vermorel, advances in AI now support asynchronous analysis, fostering comprehensive reports surpassing past iterations of LLMs.
Vermorel paints a portrait of LLMs as effective and efficient tools, armed to identify substantiated vendor claims while discarding weaker ones. Doherty echoes the advantages of this AI propensity through easy prompt creation, yielding unexpectedly detailed market insights. Vermorel extols the virtues of AI-driven research—execution at lightning speed, scalable economic research unprecedented by traditional firms.
However, while acknowledging the promising capabilities of AI, both Doherty and Vermorel recognize the technological bias inherent within these digital tools, juxtaposed against the pitfalls of market studies conducted by inexperienced analysts. AI, they argue, encapsulates a broad understanding, hence producing more encompassing reports unlike methods limited by human specialization.
Doherty ventures into a scenario involving aviation systems, exploring how vendor feedback integrated into LLM prompts could enhance market research pertinent to serial inventory management. Vermorel delves into the potential of tailoring findings based on vendor insights, highlighting how indispensable features like individual unit tracking can become focal points within cost-effective AI-driven reports.
Concerns emerge about the quality of input data affecting LLM outcomes. Vermorel maintains that even poor-quality data can yield sound results through LLMs’ ability to scrutinize and compare existing information. Superior engineering teams, he posits, inevitably produce better online content, skewing LLM assessments favorably. Despite potential biases, AI’s rapid, iterative capabilities offer rich detail faster than traditional methods.
The discussion introduces LLMs’ reliability amid concerns about inaccuracies through “hallucinations,” prompting Vermorel to clarify that while errors exist in facts, LLM-assisted online information retrieval positions these tools to outpace human error during deeper research.
In dissecting the redundancy of market consulting firms, Vermorel contends that many firms may face obsolescence unless they redefine their technical prowess beyond AI—which remains rare and expensive. Nonetheless, the irreplaceable human element consultants provide in corporate support remains a vital consideration.
Doherty questions the legitimacy of consulting firms’ proficiency, hinting at historical exaggerations. Vermorel acknowledges the existence of experts, albeit few, grappling with the breadth necessary to accurately assess diverse enterprise software. He questions the realism involved in consultants mastering varied domains consistently.
Turning to innovative collaborations that unlock AI potentials, Vermorel comments on the discord between AI’s capabilities and existing consultancy business models, where rapid, AI-driven workflows subvert traditional revenue streams. Doherty reflects on the enduring favor businesses have for established brands over technical excellence, illustrating the strategic distinction between conventional success pathways and the emerging AI-driven alternatives.
The dialogue rounds back to effective communication, with Vermorel advocating for LLM-driven research experimentation. While these AI-led insights offer efficient, cost-saving potential, Vermorel concedes traditional methodologies also hold value, particularly in adversarial research contexts where deeper probes into claims are undertaken.
Closing on a note of gratitude and reflective appreciation, Doherty concludes the episode, thankful to Vermorel for sharing his profound insights. Amidst the shifting tides in market research landscapes, Vermorel’s perspectives invite thought-provoking contemplation on integrating AI’s transformative potential with traditional, human-centric analysis frameworks, signaling an era of possibilities poised to redefine supply chain intelligence.
Full Transcript
Conor Doherty: Welcome back to LokadTV. Identifying the correct software vendor for you is tricky and often requires quite a bit of market research. Now, historically, market consultants have helped people navigate this tricky terrain.
Joannes Vermorel: Unfortunately, according to Joannes Vermorel, not only are there myriad potential conflicts of interest, but thanks to advancements in AI, that might not even matter because the era of the market consultant might well have come and gone.
Conor Doherty: Now, as always, if you want to support what we do here, like the video, follow us on LinkedIn, and subscribe to our YouTube channel. And with that, I give you today’s conversation on supply chain market research with Joannes Vermorel. So, Joannes, thank you for joining me again.
We’ve discussed in roundabout ways before, we’ve discussed previously the idea of the efficiency and even the ethics of traditional market research, and you have expressed a good degree of skepticism, I think, to put it mildly. Now, before we get any deeper into how AI might influence the future of supply chain market research, can you give an overview of what your problems are with the current state of market research?
Joannes Vermorel: Ah, so I guess we have three hours to go through the rabbit hole, or another analogy would be a matryoshka—you know, you have a problem, you just open the thing, and then there is another thing inside, and it’s another problem, and you keep unpacking more and more problems.
Rather than traditional, I think I would use the term mainstream market research, and it’s just as dysfunctional as the rest. What are the big problems at a very high level? We have incentives that are completely bogus. So, long story short, market research firms are paid by vendors. Their incentives naturally are to do what’s best for their clients, and their clients are nowadays, I would say, at least 90% enterprise software vendors.
Conor Doherty: So, you say there’s direct and indirect clients there. Like, you’re saying the vast majority of their income comes from vendors?
Joannes Vermorel: Yes, not from companies seeking guidance. So, in this situation, you end up with infomercials. Again, if you have the quasi-totality of your revenue that comes from enterprise sector vendors, then you are serving them; it doesn’t matter, you know, the incentives are just too strong. That’s, I would say, the problem at a very high level.
First of all, the fact is that the average analyst is not super savvy when it comes to software technology. In between, we have plenty of other problems, but those are really the extremes. So, we have wrong incentives in the hands of people who ultimately lack skills, experience, and even basic understanding of software, and you end up with documents that are very underwhelming. That’s, in summary, what is wrong with mainstream market research.
Conor Doherty: Is this limited to you, or do you think this is quite a common view these days?
Joannes Vermorel: I think that people like me who have software vendor experience have seen the systems a little bit from the inside. I would say generally agree. I don’t think that any of my peers look at those reports and think, “Oh, that’s very interesting; I have discovered something interesting or relevant about my peer.” If market research was done right, it would be of extreme interest to analyze your own competitors.
My own take is that the level of analysis, even for the biggest market research firms, is below what I would expect from an intern at Lokad. It’s bad; it’s very bad.
Conor Doherty: You’re talking about, for example, quadrants and being demarcated into like, “This one’s a leader, this one’s not.”
Joannes Vermorel: Yes, and again, that reflects, as I was saying, a matryoshka problem that at the very end of who gets to write those reports. These reports are not produced by visionaries or people of incredible talent but by people who are acting like journalists—in the bad sense—in the field of enterprise software technologies.
Conor Doherty: Well, it seems like off the bat, multiple things can be true simultaneously. For example, you might have consultants whose direct clients are vendors who recommend those vendors, but those vendors might still actually be the best solutions on the market. So, are you saying that the state of research is terrible, and the results are terrible due to conflict of interest?
Joannes Vermorel: Yes, but unfortunately, the list of issues is so long. The methodology is usually completely bogus precisely because the people—there are a tiny few exceptions, like the Brightwork research with a guy called Shaun Snapp, who is doing very high-quality work, but it’s extremely rare.
When you have people doing this market research who do not understand how those software technologies are engineered, then you have a very superficial methodology that takes the claims of the vendors pretty much for granted, and you end up with reports that are extremely shallow, infomercials for the vendors.
A proper methodology would involve challenging in-depth the technical argumentation of every vendor, but you need someone who can do this sort of challenge. Frequently, when you read those market research reports, the methodology never actually challenges the reality of the claims made by vendors.
So many things go wrong—the case studies, they would take for granted the case studies produced by vendors, arriving with a conflict of interest, so they cannot be trusted, etc. It’s a very long list of problems.
Conor Doherty: Because you mentioned, I believe it was Brightwork, what is it about that man’s methodology that differs from the mainstream you just described?
Joannes Vermorel: It starts with one person who is actually a decent software engineer, someone who understands how enterprise software works. The first step is producing a report with someone smart enough to do the job. If you don’t, you have all sorts of problems with the report at the end, no matter which methodology and incentives you have.
Because this person is fully competent, he can start having a methodology that makes sense, such as doing technological assessments, looking at what the technology is worth on its own. Does it have merits? Is it well engineered? How does it compare to alternative similar technologies?
If you want a product for enterprise—a solution in enterprise software is very complex, made of many pieces—look at the different pieces and assess how each piece compares with state-of-the-art alternatives. You need a divide-and-conquer perspective, which requires an understanding of the technology.
That case is a fringe case, an exception that confirms the rule that market research firms are relatively bogus for the most part. This tiny firm has a credible mission statement that they are not taking money from vendors, which solves a lot of problems.
Market research can be done right, but mainstream market research is dismal.
Conor Doherty: How does AI and, in particular, LLMs fit into this equation? Because I know you’re quite a fan of them and their applications to market research.
Joannes Vermorel: Yes, that’s something that came up quite recently with the release by a series of AI vendors of deep research capabilities. From two years ago, when tools like ChatGPT were first released, those tools were released with limited web search capabilities, and they were not very good. The problem was LLMs, large language models, are slow, and if you want to have an interesting task done on the web, you can’t do it interactively. That means you can’t ask, “ChatGPT, give me a comparison of the most relevant inventory management software” and activate the search option. Because LLM tries to give you an answer within, let’s say, 30 seconds, which means that realistically, the model can only look at like three, four web pages. This is not even close to being enough, and so the results were mostly garbage.
Now, a few weeks ago, OpenAI released a deep research mode. Some other competitors already had this mode. Google has already with Gemini its own deep research mode, and the idea is simply you give up on the interactive nature of the response. So the LLM is going to do things asynchronously, and you will be coming back. You can come back half an hour later, and then your report is ready. What is happening under the hood, it’s like a specialized agent. It’s just something that tells the LLM, “Okay, you have been tasked to do this deep research on the web. First, you need to organize a collection of web search through a search engine, let’s say Bing, where you’re going to collect the pages that are relevant.” Then you’re going to analyze every single page to see if there is something of value in this page, and then finally, you will gather all those partial analyses into a synthetic report that addresses the question or the task of the user. It works beautifully well. I was positively surprised. It is basically the same LLMs but with the tidbits of automation that make it really worthwhile.
And as a rule of thumb, let’s say OpenAI, when you do deep research, is going to check something like 50-60 pages. So it’s quite consequential, and I suspect it probes a lot more but dismisses a lot of pages. So I would say it probably probes something like 200 pages. Out of those pages, it discards the majority of them as not being sources sufficient for the analysis.
Conor Doherty: If you direct it to be sufficiently robust.
Joannes Vermorel: No, no, no, it does that on its own. So, you see, it’s just the calibration of what they have done. It seems to be at this point, pick 200 pages or so that look most relevant, keep no more than, let’s say, 50, and then produce a report by going with an in-depth analysis on what you found the pieces that are most relevant on those pages. Maybe, you know, five years from now, it would be 2,000 pages and 200x, so there is a limit. You can see that it’s the context window size of the LLM at play.
I suspect that that is a major factor still in how many sources you can collate for your final report. But if we go back to market research, these things work beautifully if you add some extra instructions to make it work. In particular, you have to provide the LLM with some guidance because the models right now tend, when you use them out of the box, are very naive. When I say the models, I mean the LLMs are very naive when it comes to dealing with enterprise software.
So you need to have a prompt that adds a lot of elements to the methodology so that you have a halfway decent report that is produced. It’s actually very straightforward. You have to add qualifiers like, “I want a maximally truth-seeking report, no marketing fluff, be extremely skeptical, do not take for granted any claim made by vendors. Make absolutely sure that every single claim you assess as a positive element to the solution is substantiated, that you do not dismiss all claims that are just naked assertions.”
Exactly like or just claims that are obviously produced with a massive conflict of interest. For example, if a vendor says, “For an inventory management software, we gain 50% productivity”—
I mean, that’s exactly the sort of claims that you should be very, very wary of. When you look at what was your baseline, you know, according against what? Against pen and paper? That’s not a good baseline. Plus, when it comes to case studies for enterprise software, the client has just as much interest to claim that there was a massive profit because then whoever was a manager in charge of the project looks like a hero, and it’s good for his or her career advancement.
So do not assume that because the claim of benefits comes with a client that it doesn’t have bias. It does carry just as much bias, if not more, than when it’s just the software vendor. So bottom line is you need to, in the prompt, say, “Be maximally truth-seeking. Be extremely skeptical. Dismiss claims of benefits where you cannot understand the rationale.” The fundamental thing is that if the vendor claims to bring benefits, is it something where they also explain how they do that, how they end up with this measurement? If they do, you can have some credibility to a study. If the claim is just because I said so or because my client, who happens to be my buddy, said so, then no. You see, the prompt, finally, it’s not very complicated.
It’s just also you need to add a few safeguards usually if you have any experience in the domain for red flags or claims that are just insane. For example, if you, in the prompt, you would say anybody who claimed that they can do 50% inventory reduction is a clown. It’s not possible, not even close. So please, in your report, degrade your assessment of whoever is making those crazy claims because it’s not in their favor. It’s like a red flag demonstrating incompetence.
Conor Doherty: Also, not to jump in, but I do have to point out that, again, anyone listening to that who is familiar with the space is probably thinking the same thing as me right now. You’ve basically disqualified everything that’s ever been printed by anyone at any time in the history of supply chain, because even if you missed 99% of those trip wires, I’m sure at some point you hit one of them.
Joannes Vermorel: Yes, and then it’s a proportion again, and LLMs are quite good at capturing that. You see, you say it counts as a negative. And then the LLMs—it’s very interesting when you see how those sorts of reports are formed—they would come with an assessment, “Oh, this vendor, oh, it seems that all they ever say is pure unsubstantiated marketing claims, and the vast majority of them are just nuts in this case.” But they often rank, so it’s like, “This was metrics, this was the best one, this was the weakest because it had these unsubstantiated claims.”
Exactly. And so, it’s important to give, also as part of the prompt, some hints on how to detect the positive and the negative. And what is interesting is that the negatives work very well. So list the sort of things that seem very opaque, claims that are incredibly vague, that are just too good to be true, etc., etc. Again, you may have to iterate; your mileage may vary a little bit. But within something like 20 minutes, you can have a prompt. You know, we’re not talking about a 10-page long prompt, the sort of prompts that I’ve been using were like quarter-page long. So it’s something that you can produce in 20 minutes, and then you’re good, and you will have a market study on pretty much any topic, by the way. And it works. I would say what you get is already vastly better than what professional market research firms are producing or even consultancy firms are producing, too.
Conor Doherty: Well, at minimum, it would also be cheaper. Even if it was of indistinguishable quality, which is not what you’re saying, but even if it were, it’s still faster and cheaper.
Joannes Vermorel: Yes, exactly. I mean, the beauty of it is that you get your report within 20-30 minutes, and you already have like a 20-page very detailed report with the citations that you, with citations, exactly, pointing out where the information is coming from. And it’s a huge amount of work. That’s the interesting thing is that with these deep research capabilities provided by the, I would say, AI specialists nowadays, you can do in hours what would have taken weeks for an assistant to compile. And that is very impressive, and the quality, again, I was—that’s where I say it’s very impressive—is that if you frame the problem as be maximally truth-seeking, yeah, LLMs, people can argue they have bias, but they do a fairly decent effort at—that’s my experience—at being truth-seeking. It doesn’t mean that it’s perfect, certainly not, but it’s quite good. It is quite good. And I would say maybe not above human, but quite—I mean, there’s trade-offs. Again, how much are you willing to spend? How much time do you want to invest?
I was saying the problem is that most of those reports are done by people who have zero understanding about the technology. The magic is that those LLMs have a halfway decent understanding of all technologies. Yes, they are not a database wizard, but they demonstrate an above-average proficiency in database design, database challenges, and whatnot.
When you’re doing market research, the challenge is that you have so many areas where you just know so little. The beauty is, when it’s done with an LLM, the LLM is almost never completely ignorant of any area of the business that you’re looking at. That makes those sort of reports very complete. I’ve been playing a lot and I’ve been overall quite impressed with the quality of stuff that can be produced. It is a little bit mind-boggling.
Conor Doherty: Well again, just to try and contextualize this for people listening perhaps with an example. Historically, I know it’s an example I come back to many times, but your distinction or your categorization of the types of enterprise software between records, reports, and intelligence systems of records—ERPs, systems of reports with BI tools, and then systems of intelligence with decision-making software. You made the point that to excel at one, you sacrifice the other—you can’t be Superman at all of these things.
Now to the question: if you were conducting a market research study using an LLM, would it be able to tease apart claims? For example, an ERP vendor who says, “Our ERP system is fantastic at handling records, and we can also do incredible forecasting and decision-making optimization,” which you know as an expert to be essentially outlandish nonsense. The quasi-totality of people would not know at first glance that structurally, in terms of the design of software, you can’t actually do all of those things brilliantly.
An LLM could tease that apart, I would say, essentially for free.
Joannes Vermorel: Probably not. It’s not again, but if you happen to be aware of this classification that I introduced, you can tell the LLM about it and then it will enforce your intent. You can just say, “By the way, I’m looking for a system of records that strictly eliminates the other things.” You can even provide a link to the page and say, “Please take into account this classification in your assessment.” That would make it more thorough.
But I would say that is the sort of thing where you don’t even really need that because my suggestion is that LLM market research is just the first step to go super fast. My recommendation is still to go for adversarial market research, where you ask the vendors themselves to explain who their peers are, what they think is good or bad with their peers, and who are the peers they respect the most, and if there are any gotchas in their view that need to be taken into account.
You can do your market research in 30 minutes, realistically two hours, and then you will get your first solid—you are not even staring at the computer for those 30 minutes. You write the prompt and you might ask one or two qualification questions like, “How do you want this? Do you want it as a report?” Then you come back, go have a coffee, whatever.
You will need a little bit of time to digest a report, so let’s say two hours. Then you switch to adversarial market research.
You shoot a few emails to those different vendors, just pick like three, and those vendors will come to you with—that’s part of the methodology of adversarial market research—with a few insights on how to think of your problem.
If there is a vendor like Lokad that comes to you as saying, “Beware, the problem you’re looking at is in fact several complementary products but products that are very distinct,” and if the rationale convinces you, then you should just repeat the market research using the LLM again.
Saying “Okay, I have this piece of understanding that was provided by a vendor, but it’s not something that is completely neutral; it’s just an understanding of the market. It’s not a piece of understanding that is just favoring this particular vendor.” You can also take the reports that have already been designed and have them evaluated.
For example, let’s say you are a company that has aircraft parts and you want inventory management. Then the first vendor that you contact tells you, “Beware, you need to have a system that supports serial inventory management.” That means it’s not about keeping ten units in stock; it means that for every unit in stock, there is a certain number and you need to keep track of that.
You have inventory management systems that support serial inventory and those that do not. Among those who do support inventory management, some manage that as a first-class citizen, so it’s supposed to be the primary use case, and articles that have no serial number are like the second-class citizens.
You have other inventory management systems where it’s the opposite. A vendor can tell you that it’s a very important feature if you want to have serial inventory management. That’s completely different compared to just regular inventory management.
Fine, a vendor tells you that, and you didn’t know. So now I’m going to do again my research on inventory management systems saying, “Hey, I really have to pay attention to this serial inventory capability. It is critical because I am doing aviation.” Redo this market research with this criteria in mind and regenerate the reports.
I see LLMs as very complementary to this adversarial market study. As you get some tidbits of feedback from vendors, you can just incorporate those extra insights on how you should even look at the problem into your LLM prompt and just rinse and repeat, regenerate. It’s cheap, relatively fast, and you will get something that will be increasingly tailored to your use case.
Conor Doherty: Certainly, that sounds good in theory. But don’t you run into the pre-existing wall, which is LLMs are trained on data sets? You’ve already decried the state of the current literature in supply chain, so if you wanted to run a market study using an LLM and you give it all of these very careful parameters, nudges, and qualifiers, it still has to read the information that’s publicly available. If the publicly available information is garbage, isn’t that garbage in, garbage out?
Joannes Vermorel: No, I mean, that’s again. Here we are back to the challenge of LLMs: one of the most incredible things is that LLMs are garbage in, quality out, which is very weird. It’s why I think it took a long time for even the software community to discover these things; it is fully counterintuitive and goes radically against what was the dogma ten years ago, which was garbage in, garbage out.
It turns out that when you inject the entire web into those models, you get something that is very decent, which is strange. I mean, when you think that ChatGPT has ingested all the nonsense of Reddit and it remains sane, it’s quite remarkable.
Here, I would say no, it works. The challenge is that yes, vendors have sometimes materials that are of very low quality, but again, an LLM will use that to do a comparative assessment.
If everybody is equally bad, then it’s difficult, but very frequently what happens is that some vendors are way worse than others. Maybe even the top vendor is not your dream vendor; it’s a little bit lacking. Someone has to be the best.
Exactly, and my casual observation is that when it comes to quality of products, quality of software products, the quality of technologies, there is an immense correlation between the quality of the materials published on the website of the vendor and what goes on under the hood.
Companies that have decent engineering teams have content on their websites that explain what the product does, how it does it, etc., and it is also decent invariably. Conversely, if the team is small and it’s outsourced into some cheap, underdeveloped country, then surprise, surprise, the quality of the website and the technical content is also dismal.
No surprise, those sort of things tend to go hand in hand. That’s what makes the LLM case for market research so powerful, at least as far as software is concerned. You have this immense correlation between the quality of content online and the quality of the product; they really go hand in hand. I don’t think in my entire career I’ve seen any exception to this rule.
Conor Doherty: Okay, and not but, and it does occur to me that there’s a possibility here that what you’re describing and certainly what is already available might, in fact, yield an objectively better market study on the entire suite of available options, but the question then becomes: is that necessarily what the end user, in this case client companies, actually want? For example, I’m sure that, to play out this thought experiment to play it out, um, you conduct Joannes’ company, you have a company called Lokad and you want to find a vendor and you do a full market study and there are 10 options, nine of them are the big boys we all know the names, and then number 10 is Conor’s ERP.
And Conor’s ERP is the best one, but no one’s ever heard of it, but it is, along with your metrics, the best option. He’s got the best software, he’s got all the best public documentation, he explains how everything’s done, it’s fantastic. You then show that internally and they go, “Who the hell is Conor? Give me one of the big names.” I mean, because people want big names.
Joannes Vermorel: No, I mean, first, again, LLMs are surprisingly good. You know, it’s a no-name company. What qualifies a no-name company if it has like very little materials, very little documentation, very few features, very few everything, age, prestige clients, yeah, but again, that will be reflected in terms of the depth of what they do, you know.
If you look at OpenAI, it was, until very recently, a no-name company, correct? It completely exploded in 2023. Before that, it was a very obscure, heavily funded but very obscure company of the Silicon Valley. So you see, my take is that again, LLMs are fairly good at this sort of assessment, and they will take that into account. They will also take that into account that and thousands of other things.
So don’t think that when I say be maximally truth-seeking and whatnot, that LLM is just going to be an idiot and suggest something like a super, super obscure provider coming from Bashtoan or whatever. This is not what you will get out of those. Now, if we get with your preconception, if you already know what you want, then don’t use an LLM to justify why.
You know, that’s wishful thinking. You know, if you assume that you want to do a genuine market research, then you should actually suspend your preferences for certain vendors. You know, otherwise, why do you want to even do this market research? If you’ve already decided that you want to pick a vendor, then pick a vendor. Pick this vendor directly and spare yourself the sort of fake process to kind of justify the decisions that you’ve already made. You know, you will just save time and money to a company.
So you see, if you’ve already decided and then you do a market research, this is nonsense. There’s no logical argument to support this sort of thing, so you need to approach the problem from at least a perspective where you’ve not decided yet and what you’re trying to do with this LLM is just to have a maximally truthful, objective statement.
And that’s, I believe again, in this regard, those tools are quite efficient, and if you’re really afraid those tools have limitations, also, if you think that the LLM is just missing a vendor that it should be looking at or whatever, you can just provide. You can literally tell, “I want vendor A, B, C, D, F, and whoever, in addition to those, thinks it’s most relevant.”
So you see, there are no hard rules here. You can literally tweak your prompt, and the LLM will just adjust the composition, but just try to do it in a way that doesn’t introduce a massive bias for the LLM. So you see, just do it in a way, for example, if you want to explicitly list the vendor, you have to be careful not to hint who you want to win the process.
Because if you prompt the LLM with, “I feel that this is the best option, exactly do a case study, do a market research study on inventory management software and don’t forget this one vendor which I really think is the number one,” you’re introducing a massive dose of bias. So here you just—but again, you don’t have to be extremely smart to figure that out. It’s just about phrasing your own prompt in a neutral way. Otherwise, you will get bias out of your report.
Conor Doherty: So Joannes, I think it’s worth just planting a flag here. I mean, we did it to one degree earlier, but just to really sketch it out. When we talk about the skills, the digital literacy skills required to produce this kind of information, it’s not advanced. We’re not talking about writing in Python or anything like that. This is basic stuff, right?
Joannes Vermorel: Yeah, I think if some people decide to introduce a prompting diploma or prompting certification for being able to prompt an LLM, that will be the sort of things that you can master in two days. You can have your prompt cert, prompting certificate in two days. Yes, it is not difficult. It is absolutely straightforward.
And again, you can even interact with a tool to figure out what is wrong with your own request. So it’s a fairly interactive process that also gives you feedback on how to improve yourself if you ask for feedback. Well, the second point then, because the first was like the skills required to do this, but the second point, and it was something that you mentioned just a few moments ago, was reliability.
Now you didn’t mean it in this context, but it does tee up a concern that I’m sure some people have when they listen to this, which is historically, LLMs’ reliability has been called into some degree of dispute. So, for example, hallucinating, or I know you like the term confabulating. You ask an LLM to do a thing, “Hey, find out this piece of information,” and it doesn’t want to tell you, “I actually don’t know,” or “There is no information,” and it just confects information mostly out of thin air.
How plausible or valid a concern is that in the context of deep research, which of course is not the same model as people typically think of when they think of LLMs?
It is the same model, but the thing is that, as a rule, when it does deep research, the LLM is prompted to go and fetch the information on the web. So here it is again, LLMs are not databases for facts and knowledge nuggets. They were trained on that, they were trained, but if you ask, “What is the exact altitude of Mount Everest?” They will kind of remember, but if they can actually look up the information online, it will be much easier for the tool to be absolutely sure.
So in my experience, the amount of confabulation or hallucination when using OpenAI deep research is fairly low. You can really see that the model is leveraging the information that exists on the page to build its own assessment, and thus you’re not asking the model to just kind of invent or remember all that it knows about obscure vendors. It doesn’t even try.
It would just say, “Okay, here is the list of pages that I’ve retrieved from this vendor. What are those pages telling me?” That’s how it works. It is still possible for the LLM to make things up, but I would say to a much lesser degree than an actual human. You know, an example of mistake would be on the page of the vendor it says, “We have over a thousand clients,” and then the LLM in the report would say, “They claim to have 1,000 clients.”
It’s a subtle approximation in one case they were saying “over 1,000,” it was distorted into exactly 1,000. Okay, again, that’s kind of minor distortion in my own experience. The distortions are infrequent, and when they exist, they are relatively on the more insignificant side, as opposed to, for example, inventing an entire vendor out of thin air and just attributing qualities and weaknesses to this vendor out of thin air. This will not happen.
This will not happen. 100% of the vendors will have a website with sources to be listed and everything. I’ve not seen a report to invent stuff out of thin air that was extravagant. It did happen, but it was subtle and sometimes you’re even on borderline on inference. You know, it looks like they have this, and you just take a shortcut and say they have this, but it’s not completely clear. When you look at the documentation, it’s kind of implied, and the LLM took the bait.
Conor Doherty: Well, yeah, of course. Well, again, even if you paid a consulting firm to produce a report, you would still presumably read and evaluate it yourself. So it’s not as if the claim here is just use deep researchers’ output and take it at face value that it’s absolutely perfect. You still have to interrogate yourself.
Joannes Vermorel: To interrogate yourself, yeah, exactly. I mean, again, you can’t suspend your judgments. You need to make use of that. And again, I think the thing where those tools really shine is that you can iterate at a pace that is just simply impossible, even with the best consultants. It’s like, you could repeat a one-month study—something that would take one month to a human—but done by hand in 30 minutes. It is extremely impressive. I mean, again, we are talking about producing a 20-page long report, super structured, completely tailored to your request in 30 minutes. It completely changes the game when it comes to your capacity to iterate.
You can effectively end up doing 20 market studies in a row, each one iterating on the previous one so that you’re more and more focused on what you actually want.
Conor Doherty: Well, you spoke a moment ago about making inferences. Is it reasonable for anyone listening to make the inference that you see the role of the market consultant and consulting firms, etc., either becoming redundant or having already become redundant as a result of this technology?
Joannes Vermorel: If we are talking in terms of deliverable and you assume that the deliverable is a market study, then indeed many, if not most, market research firms and consultants delivering this sort of service are indeed made completely obsolete, unless they can really prove that they are able to do better than a machine. It starts by having people vastly capable at a technical level to produce an analysis that is smarter than the LLM can do.
It does exist, but it’s going to be super niche, just because there are not that many people of this caliber in these sorts of industries and slower and more expensive, presently. Yes, we are talking about something where yes, it’s going to be much slower. Those software vendors are not buying market studies from them; what they are buying is advertising. So as long as enterprise software vendors are willing to spend money on those market research firms, market research firms will do just fine, which has nothing to do with the existence of LLM capable of producing market research studies.
That’s not what those enterprise software vendors are paying for; they are paying for the advertising. It is just a channel. They could alternatively spend the money on Google ads; they just decide to spend it on some market research firms or a fraction of their budget on this area.
For consultants, my take is that very, very frequently, consultants—what companies or actually executives are really paying for—is companionship or moral support, and this is not something that the LLM will give you. It seems a little bit dumb when you put it that way to say, “Oh, why did you pay this $100,000 mission?” “I was feeling lonely, I was feeling insecure, I needed to have someone to have my back, so I just took them. They’re good guys; they support me.” That sounds a little bit ludicrous, but yes, that’s, I think, to a large extent, you know, that explains this sort of attitude, explains most of the consulting business.
The fact that you’re buying a market study is just the pretense to phrase a mission in a way that doesn’t sound as silly as “I’m looking for a corporate coach.” But I mean, beside that, so that’s why I think that it might not change again. The existence of LLMs and their capability may not change so much in this regard, because ultimately this was not what was being bought already. The report was a pretense for something else. It was not what justified, truly in the eyes of the management, paying the consultants in the first place.
Conor Doherty: So again, reading between the lines, having listened to all that, would it be reasonable to say that you don’t see market consulting firms having valuable expertise to directly apply to the researching the market? So their expertise does not lie in that domain, even now or even historically, you might even argue, but you’re very skeptical a consultant—a professional consultant—cannot bring valuable expertise, that intangible quality, to bear on this out of 8 billion humans and probably a million, you know, consultants worldwide.
Joannes Vermorel: Yes, sure, there must be dozens, hundreds of people who are capable of doing that. Now the question is that how many truly are capable of doing that? What is the percentage? My take is that the percentage of people who are able, among consulting circles, to do this sort of assessment as far enterprise software is concerned, better than LLM are just vanishingly small.
And ethically or devoid of bias, yeah, exactly. I mean, both. But I believe that the bias is more for market research firms. For consultancy groups, I think most of them don’t have the problem to the same magnitude; they do have the problem, but it is a relatively small problem. The much bigger problem is just the skills and competency to even do this assessment in the first place.
And again, this is because as a consultant, you keep doing things that are so incredibly diverse. You know, it is very difficult. Today, you’re being asked to be an expert on inventory management software; the next day, you want to be an expert on yield ratio for a chemical production line. The amount of diversity of missions is just staggering, and so it is not very realistic that you will have people that are very competent in these sort of things.
Conor Doherty: So, Joan, it’s interesting because listening to you and this discussion on market consulting firms, I recall about a year and a half ago, two years ago—and again, correct me where I’m wrong—although it was me who actually wrote the paper, we reviewed a paper by Harvard Business School, I believe it was called “Navigating the Jagged Technological Frontier.” And it was interestingly produced, if I recall correctly, with BCG Boston Consulting Group.
In it, and I’m going to summarize it massively, and point out anyone can comment where I may get it wrong, again, I’m recalling in real time, people who received training with LLMs or generative AI performed better at certain tasks, both qualitative and quantitative, than people who did not have this training. So my question to you now is, anyone listening to this could just as easily say to you, “Well, Joannes, if we just give these tools that have got better in the intervening 18 months to highly trained consultants, won’t they just produce the world’s greatest consulting reports?”
Joannes Vermorel: Yes, they would. I mean, certainly in the hands of consultants, you could produce fantastic reports. But now the question is that it’s a business model of consultancies. You cannot charge a large consultancy firm like BCG or McKinsey for two hours of work, so that challenges a little bit the business model. But again, if we go back to the idea that the report is really what is being acquired, I really challenge this assumption.
My take is that it’s not what is being paid for, so the availability of technology is irrelevant because it’s not what is being bought for the quasi totality of those consulting missions.
Conor Doherty: One of the things I would actually tend to agree with, in the sense that the longer I’ve worked in this space and the more I’ve listened to, the more I’ve spoken with professionals at trade shows, at conventions, the more I realized that big-name endorsements is the impimeter that people generally look for. As a way of sort of wrapping up the conversation a bit, there’s a phrase, “Better to fail conventionally than succeed unconventionally,” that in big companies, there is the perception, “Well, I don’t want to be the company that takes a chance on this obscure company that might be, on paper, according to either a consultant or an LLM, the best option. I’d rather just stick with the shorter menu of established successful names.” So your thoughts on that?
Joannes Vermorel: I am, again, everything is relative. Most enterprise software companies are not brand new; even if we’re talking Lokad, we are talking about a company that is 15 years old. It’s not exactly super young. My take is that the problem is more like you need to have something that is kind of convincing.
And the typical problem that maybe upper management would face is that they end up with messages from their subordinates that are completely unclear, and writing skills might be subpar, problem analysis might be subpar. The consultants provide a nightmarishly long PowerPoint of 80 slides that doesn’t make head or tail of sense or makes little sense, and it is such a headache for the top management. They have to make a decision, and so you fall back on something that feels not too risky.
But again, I don’t think most people who manage to get through the ranks of high-ranked executives are not idiots. If there is something that is convincing, that is kind of neatly explained, they would just go along. It’s not, and where I think LLMs are making something that is game-changing is the capacity to produce a very high quality rational in writing to support the case of this option versus this option.
This is something that, if I see most large companies, the typical writing skills are fairly low. Some companies, like Amazon, are notoriously known for managers being able to write five-page memos of high quality. It is very rare, and I think a lot of the success of Amazon boils down to this capacity to tackle problems in writing, not powerpoints, and really think in depth about something. Here, the interesting thing with LLMs is that it suddenly makes this exercise much more accessible to people who do not have all those writing skills. Obviously, having the writing skills will make you even better, but overall, it’s just an immense facilitator.
Conor Doherty: So then, as a closing thought and a piece of advice, you would advocate leveraging LLMs for, at minimum, creating more meaningful communication?
Joannes Vermorel: I would say if you are thinking about anything that is like market research, start with these deep research capabilities with half a dozen AI chatbot vendors who support this deep research capability. This will be most likely a groundbreaking experience for you, and you will realize that you can do in hours what would otherwise have taken months and cost a lot of money as well, and be very cheap.
Yes, it will be very cheap. That’s that; just see for yourself, you will be pleasantly surprised. Then the next step afterward is just to fall back on advisory of market research, which is also very lightweight, but we are talking about something that would take maybe several days, not just several hours.
Conor Doherty: All right, well Joannes, I don’t have any more questions. Thank you very much for your time and for sharing some good insights, and thank you very much for watching. We’ll see you next time.
