00:00:00 Introduction to startup tech audits
00:01:42 How VCs led Joannes into tech audits
00:03:24 Tech audits as a revenue stream
00:05:08 Evaluating software in startups
00:06:36 Joannes’ experience in machine learning
00:08:50 Key aspects of a tech audit
00:10:02 Security vs. correctness trade-offs
00:11:46 Writing investor reports effectively
00:14:57 Evaluating startups beyond checklists
00:17:04 Adapting audits to business context
00:20:24 Interviewing CTOs and reviewing code live
00:24:12 Assessing developer familiarity and leadership
00:28:22 Judging developer competency
00:30:49 Red flags in code quality
00:33:47 Insights from commit history
00:37:03 Cloud cost sustainability
00:42:14 LLMs disrupting NLP startups
00:46:29 Finding lucrative software niches
00:50:32 Risks of technical debt
00:55:34 Over-investing in scalability
00:59:59 Clear writing signals good engineering
01:04:08 Why tech audits are rare in investments
01:07:12 Closing remarks
Summary
In a recent LokadTV episode, Conor Doherty interviewed Joannes Vermorel, CEO of Lokad, about his decade-long experience conducting technical audits of startups. Vermorel’s “lightning tech audits” offer venture capitalists expert evaluations of technological stacks, diverging from traditional methods by focusing on dynamic, context-specific assessments. His process involves intensive, one-day evaluations, including interviews with CTOs and engineering teams to gauge code quality and team functionality. Vermorel emphasizes the importance of passion, clear documentation, and wise resource use in tech development. His insights reveal significant industry oversights and highlight best practices for both investors and tech entrepreneurs.
Extended Summary
In a recent episode of LokadTV, Conor Doherty, Director of Communication at Lokad, sat down with Joannes Vermorel, the CEO and Founder of Lokad, to delve into one of Joannes’ lesser-known but highly impactful activities: performing technical audits of startups. Over the past decade, Joannes has conducted these audits, providing venture capitalists with expert opinions on the technological stacks of potential investments. This practice, which Joannes refers to as “lightning tech audits,” has given him a unique vantage point on the best and worst practices within the tech industry.
Joannes’ journey into technical audits began in the early years of Lokad when he was contemplating bringing in venture capitalists. Although he ultimately decided against taking formal investments, his interactions with investors led him to offer his expertise in evaluating the technological aspects of other startups. This pivot allowed him to transform what could have been time-consuming meetings into valuable sales opportunities, offering his services to venture capitalists who needed an expert opinion on the tech stacks of companies they were considering for investment.
The core of Joannes’ audit process is a one-day, intensive evaluation that diverges significantly from traditional auditing methods. Unlike the standard approach, which relies heavily on checklists, Joannes’ method is dynamic and tailored to the specific context of each startup. He begins with a 20-minute interview with the CTO to understand the company’s technological landscape and then spends the rest of the day conducting one-on-one interviews with the engineering team. This hands-on approach allows him to assess not only the quality of the code but also the team’s fluency and coherence in navigating their own technology.
Joannes emphasizes that his goal is not to become an expert in the company’s technology but to assess whether the team is functional and whether the technological trajectory is sane. He looks for patterns in the codebase, the history of commits, and the quality of documentation. For instance, he can quickly gauge the quality of the codebase by looking at the consistency of coding styles and the clarity of commit messages. He also evaluates whether the team is making wise use of their limited resources, a critical factor for startups.
One of the most striking insights Joannes shared is the frequency with which tech companies can go through multiple rounds of investment without anyone having taken a serious look at their technology. He finds it baffling that even in later investment rounds, the technological stack often remains unchallenged. This lack of due diligence can lead to significant oversights, as was famously the case with Theranos.
Joannes also discussed the importance of passion and care in technology development. He believes that a lack of care is a killer for tech companies. When the CTO and the engineering team are genuinely passionate about their technology, it shows in the quality and coherence of their work. Conversely, when companies outsource their development to freelancers or lack a deep commitment to their technology, the results are often disastrous.
In terms of best practices, Joannes highlighted the importance of clear writing and documentation. Whether it’s comments in the code or the articulation of problem statements in tickets, clarity and conciseness are key indicators of a well-functioning team. He also values the use of unusual but well-suited technologies, which can indicate a deep understanding of the technological landscape and a commitment to finding the best solutions.
Joannes’ approach to technical audits is a refreshing departure from the norm, focusing on the practical realities and specific needs of each startup. His insights offer valuable lessons for both investors and tech entrepreneurs, emphasizing the importance of passion, clarity, and a tailored approach to technology development. As the tech industry continues to evolve, Joannes’ method of lightning tech audits provides a robust framework for evaluating the true potential of technological innovations.
Full Transcript
Conor Doherty: Welcome back to LokadTV. Today I’ll be joined in the studio by Joannes Vermorel, Lokad’s founder and CEO. We’ll be discussing one of his unique side quests: performing technical audits of startups. Few people actually realize that for about a decade, Joannes has been evaluating software companies, checking everything from code and infrastructure to staff and leadership itself. He and I will discuss some best practices as well as what he’s learned along the way. Now, as always, if you like what you hear and you appreciate what we do, subscribe to the YouTube channel and follow us on LinkedIn. And with that, I give you today’s conversation with Joannes Vermorel. Joannes, welcome back. As I mentioned in the introduction, we’re here to talk not necessarily about our usual matter but something that I think is still of significant interest. You do a lot of tech audits, is that correct?
Joannes Vermorel: Yes, and it has been the case for over a decade, probably almost 14 years or so.
Conor Doherty: And how exactly did this come about? Because, again, anyone who’s been watching LokadTV—what is this now, the seventh or maybe eighth year, maybe Max off camera can confirm that—they’re familiar with you talking about supply chain and all the tech involved in that. But this is probably the first anyone’s heard of this, and yet you just said you’ve been doing it for 10 years. So, as far as side quests go, it’s been a subtle one. How exactly did it come about?
Joannes Vermorel: In the early years of Lokad, I was pondering the case of taking investments, you know, bringing in venture capitalists. It turned out that I never brought formal investors into the company, but in the early years of Lokad, I talked to quite a few of them. After a while, you know, the way it goes is that you either send an email or they send you an email saying, “Oh, let’s grab a cup of coffee or something.” It takes time meeting with investors to discuss your company and whether they would be interested in investing and whatnot. It takes time, and at some point, I was thinking, “Okay, I’m spending all this time meeting with those potential investors, and that is time that I’m spending not building, not selling.” So I thought, “What about selling them something?” So, you know, just treat those meetings with venture capitalists as sales meetings. It turned out that I mentioned to those people that if they were interested—so I was not like a super hot lead for them because I was not really desperately looking for cash—but I was telling them that Lokad might not be their ideal startup because I’m not desperately seeking cash. But if they were looking for someone to forge an expert opinion on another investment, I could be their man. My specialty was not having a look at another startup and judging whether they would make money—that’s beyond my expertise, it’s very difficult. But what was well within my expertise was to have an educated opinion about their technological stack. It turned out that when you are a venture capitalist and you invest in tech companies, a big portion of the valuation for many of those companies is the technology they are developing. Is the technology truly valuable? Is it worth the pre-money valuation that the founders are asking for? And so, lo and behold, I started to do those tech audits. I’ve been doing about one, sometimes two per month for, again, I think it has been 12 or 13 years, so it has been quite a while. And so I’ve done lots of companies, and the format of those audits is very short. By the way, you were asking how do I find the time to do that. Well, I’ve always kept those audits as lightning tech audits; everything is done in one day.
Conor Doherty: We’ll definitely come back to that because there are implications to the approach. We’ll get into how your approach would differ from what most people would consider the status quo for what essentially amounts to a consultation service. However, just to properly frame this, because you’ve used the term “expert opinion” and “expertise,” to anyone who’s not familiar with what qualifies you for this post, feel free to outline your many laurels.
Joannes Vermorel: I’ve been passionate about software technologies for a long time. I started programming in primary school, spent most of middle school and high school getting better at programming with more fancy stuff, and I’ve always been extremely interested in all things software-related. At the time I started doing those audits, I was not yet 30 but close to it, and at the time, I already had almost a decade of professional software experience with my own software projects but also other projects that I had done at university and in other companies.
Conor Doherty: Now, you also worked in America for a while at AT&T Labs. It was a time when they were really pioneers in machine learning. They started doing machine learning back in the ’90s.
Joannes Vermorel: Yes, they were onto those machine learning things way before artificial intelligence became cool and hyped. There were people working on this stuff, but again, that was one part of my interest. When I audit companies, very frequently machine learning is just a tiny portion of the problems. Very frequently, it’s not even part of the problem. Those companies can be of all sorts. There are software companies, but they can be anything from a biotech company that uses software extensively, to online gambling, to a B2B productivity app, to embedded software that goes into an appliance sitting in a surgical operation room in a hospital. Software comes in an extremely wide array of situations, and the technologies that go into those startups vary enormously from one company to another, even if the core is always software. That’s where I think my expertise really lies.
Conor Doherty: When you talk about performing tech audits, I think people probably have some concept of what that is. But what exactly goes into that? What areas do you focus on? Are you evaluating just the software? Are you evaluating people? What exactly goes into it?
Joannes Vermorel: The deliverable, because we have to start with what is being delivered, is a memo that I give back to the investor. A venture capitalist is about to invest in a startup. They typically have a letter of intention that is already signed. They are about to put anywhere from 2 million to 20 million, sometimes more, dollars or euros into a company. A big part of the valuation is the technologies they’re investing in. They are investing in a team that has supposedly already developed something. I do not make those audits for seed stage; we’re talking about Series A, sometimes Series B. So we’re talking about people who have normally at this stage already developed something, which justifies a higher valuation. Now the question is, is it worth what those people are asking for? The worth is not only what they have but also the technological dynamics within the company so that they will be able, if they get the money from the investor, to complete the objectives of their roadmap. People invest not only in what the company is but most importantly in what the company will become tomorrow. I cannot assess whether their financial projections will be correct. My expertise doesn’t lie in the future of, let’s say, gambling markets. I don’t know. That’s for other people to assess. But if they show me an app for gambling, we have to check whether it’s going to be secure because if we are talking about gambling, for example, it has to be extremely secure. That’s the sort of question I can answer. If it’s an appliance that is going to be used during a surgical procedure, then correctness is absolutely vital. Quite literally, if the thing malfunctions, you can potentially kill a patient. The sort of challenges are very different.
Conor Doherty: Just to go back, when you give the example there, I understood the security one and that obviously with your jacket, but the idea of the surgical piece of equipment, what is your input in terms of evaluating that correctness? Is it going to do reliably what it’s supposed to do?
Joannes Vermorel: It’s not like a security problem because those things are going to be isolated from the network. They are going to be operated in an environment that is already very secure from an IT perspective—no network connection, these sorts of things. But the question is, how do you ensure that you don’t suffer glitches, bugs, or anything that could literally compromise the fact that the thing is going to work? Many machines now are automated. If it operates a pump pumping some kind of fluid into the patient, how do you make sure that it’s not malfunctioning, that it does exactly what you want it to do? So the bottom line is that the way I approach the deliverable is by forging an opinion. That would be a memo that I give back to the investor on whether I think that, you know, what is my general opinion on the value of this company on its own terms. And that is really the important thing. I am trying to forge an opinion on what the company is worth according to what they’re trying to achieve. If you’re developing a video game, the question is, are you developing something that is very addictive, very engaging, fun, etc.? Those are your terms of engagement. If you are developing a biotech where you have a new diagnostic technology that supposedly lets you diagnose a pathology earlier and faster than an alternative method, does it work? Is it real? Do you have some statistics that are sound? Do you have all the processes that seem to indicate that your empirical results are exactly as they should be? If we have a productivity app, the question is, for a B2B app, how do you deal with the fact that you have hundreds, potentially thousands, of features or micro-features? That’s the thing with enterprise apps; those things can be like a maze of features because you have so many cases to handle. How does the company manage all this complexity? Are they just drowning under their own complexity, or do they manage it well? Is it well organized? Depending on what sort of startup I end up auditing, the terms of engagement—what is the value proposition, what is the essence of the value—are very different. My deliverable is to forge and reflect an opinion that I give back to the investor on what matters the most. If there is anything that I see during this audit that would compromise this core value from being delivered, I point it out.
Conor Doherty: Again, you said you’ve been doing it for about 10 or 12 years. Assuming one to two a month, you’re talking about at least north of 100 performed. So, what would be the bulk? Because you outlined, well, I could be looking at a gambling app, I could be looking at medical equipment. Obviously, these verticals are very, very different. So, the bulk of your work would have been in or what’s the typical vibe when you audit?
Joannes Vermorel: It’s really software companies. It’s people who are developing software. Sometimes there is a bit of a physical element, you know, that can be people who develop software for a piece of equipment. Sometimes there is more or less third-party software to be integrated. Is it an app that stands alone, or are you integrated into a lot of other things? Plenty of situations. I mean, obviously, probably two-thirds of what I have done is B2B or professional equipment, and one-third would be B2C. But that’s as far as classification goes; otherwise, it’s extremely diverse.
Conor Doherty: Well, you’ve given us an effective top-level view of the deliverables and your background. But in terms of the major steps, how do you forge an opinion and reflect on what are the steps involved in actually arriving at that memo and that opinion?
Joannes Vermorel: That’s where my approach extensively diverges from what people usually consider as an auditing process. Just for the audience to recap, an auditor, I would say 99% of the people who do audits of any kind, have checklists. They have their long checklists with several hundreds of questions, and they would go to the company and ask, “Do you do this or that?” Yes, no, yes, no, etc. The auditor just goes through the list like a drone. My take is that this process is complete nonsense for technological audits, especially for startups.
The problem is that this list, whatever it is, is never going to challenge the startup on its own terms. It’s completely irrelevant to challenge, let’s say, a company doing gaming on whether they are super secure. If you do a video game, security is a secondary concern. In a startup, your resources are scarce. Does it make sense for most video games to invest in having a security officer and invest heavily in security features that complicate the life of the users? If you’re doing casual video games, for example, it doesn’t make any sense. Nobody cares about the fact that you’re playing Farmville.
The bottom line is that having checklists is a recipe to be a massive waste of time for the founders and a massive distraction for the investors themselves. They might say, “Oh, the startup is compliant with hundreds of irrelevant items,” and then you would say, “Oh, but they’re not compliant on these 100 items,” which are also completely irrelevant to the success of the company. My take is that they have a business case; they say they want to go in a certain direction business-wise. I let other people assess if there is money to be made. I don’t assess that. I mean, yes, I have my own opinion on whether it seems plausible, but I don’t challenge the size of the market. My take is that, okay, let’s assume there is a market and that if you execute this thing excellently, there will be people to serve, and those people will be willing to pay you.
Now the question is really, will you deliver on your premise? You are developing a technology that is supposed to do something. Will you deliver on what you’re actually promising?
Conor Doherty: And you are in a position to provide that level of insight. You can speculate on the technology and…
Joannes Vermorel: Compared to an auditor, if I go back to the checklist, my take is that when I start the day, during the first 20 minutes, I will be interviewing the CTO typically. The idea is that within 20 minutes, I need to mentally architect a series of questions to guide the discussion toward the things that are of most interest. After the 20 minutes of presentation, typically done by just a casual conversation, I conduct the rest of the day through a series of interviews where there is someone in front of a computer showing me their codebase. I conduct a series of one-to-one interviews where we are both looking at the same screen, and an employee of the company acts as my guide through their codebase.
Conor Doherty: So, you’re Virgil.
Joannes Vermorel: Yes, they are driving me through their codebase. I don’t ask for a copy of the codebase because that could create tons of problems and make me liable for intellectual property. The idea is that I want to have a look at the codebase, and that’s the way I look at the technology. But codebases can be extremely extensive. Even a small company with five software engineers working for a couple of years can easily have 100,000 lines of code or more. If I were to navigate this codebase on my own, 90% of my time would be wasted figuring my way. So, I have an engineer sitting next to me, and I ask questions, and they show me the code. We review the code together, obviously a sample of the code; we cannot review the entire codebase.
Conor Doherty: Given the potential financial stakes here, why wouldn’t you just look through this with the CTO?
Joannes Vermorel: First, the CTO is not necessarily knowledgeable about all parts of the codebase. If it’s a team of five people, including a CTO, then probably yes, the CTO is competent on the entire codebase. If you are talking about a company at a later stage with 20 engineers, most likely the CTO is not competent on every segment of the codebase. But more generally, it is also very interesting for me to have other people in the company sitting next to me because then I can see how fluent they are with the codebase. Are they lost in their own technology? When I ask a basic question like, “How do you access your persistent layer?” or “How do you access the data that you’ve recorded?” If people have no clue, it’s not good. If I ask them, “What is the last thing you’ve done?” and they say, “I’ve been developing this feature,” and they are lost when trying to show me what they’re currently working on, it is not good.
Interviewing the CTO is good, but I also need to assess whether the team is on par with the CTO. That can go both ways. Sometimes the CTO is very good, but the rest of the team is not so good. Sometimes it’s the opposite. Sometimes the CTO is very weak, but the rest of the team is actually pretty strong. It varies, and that will also be part of the opinions that I give back to the investor as part of this memo for deliveries. I also have an opinion on whether the contributors currently part of the company are up to the challenge for the next phase. If the investment happens, sometimes when a company starts, their budget is extremely limited, and they go for people who are not super talented in terms of technology. This frequently happens when the founder is not a tech founder. When the founder is someone who had insider knowledge, who knows there is a specific problem that needs to be addressed, and this person sees a need and willingness to pay. This happens very frequently in B2B markets. But this person is not technical, so they fetch the first software engineer they can find. They might not have a big budget, so their first hire might be someone whose main advantage is being very cheap.
Again, that’s okay, that’s okay. I mean, you’re starting with nothing. You have, let’s say, €20,000, you’re putting that in the company. For you, that’s already quite an investment. Thanks to those €20,000, you manage to have like a half a million seed. That’s good. You can have your team of, let’s say, two software engineers to develop the product. Then you want to go to the next stage and now have the first round of, let’s say, €3 million. Well, at this stage, if we’re talking of multi-million investment, that is a stage where now you have the money to have real talent in terms of software engineering. So the question is, do the players that you currently have are up to the challenge for the next level, the next stage of the company? So that’s also, that’s not the only question, but that’s also one of the questions that is being looked at.
Conor Doherty: Just to be clear, as part of the memorandum, the deliverable, you would potentially make recommendations on who should and should not stay?
Joannes Vermorel: Yes, yes. Again, it’s whatever would help or compromise the technological value of the company, its technological trajectory. Sometimes you have the wrong players. I mean, you had the players that you needed to win a super local championship, but if you want to play nationwide or even go for the world championship, then maybe you need different players. Bringing more money changes the dynamics. Suddenly, you can potentially afford people that are way more experienced, way more talented. It varies. Sometimes people are very talented, but for some reason, the dynamic in the company is super bad. Sometimes it can be two CTOs for some reason. That did happen in some audits. So you have two technological leads, and those two people are pursuing different roadmaps. Then my message is, pick one. You cannot remain completely schizophrenic; you need to choose. That would be a business decision because it’s different markets that you’re trying to chase. But ultimately, that’s the sort of problem where an investor needs to know if there is something wrong with the tech or if there will be something wrong in the future with the tech. Because sometimes there is no problem yet, but adding money is like adding fuel to the fire. If you have something that is slightly dysfunctional, if you start pouring a couple of million on top of it, what was a small problem may just be magnified enormously by the investment itself. So that can also be a problem. Again, there is no checklist for me, there are no rules. It’s whatever I see that makes sense in terms of problems to be fixed or recommendations for the investor. That is part of my scope. The idea is that I keep that as a one-day mission because I don’t have the time to do much more. But also because the founders themselves are very busy. I think part of the problem with classical audits is that those things are such a waste of time. It takes weeks, it completely paralyzes the company, and there is nothing good that comes out of it.
Conor Doherty: Well, on that point actually, so you’re saying you don’t use a checklist, you aim to be in and out in less than a day. When you say a day, that’s a workday. So you’re talking about instead of a multi-week process, which is multiple weeks of 8-hour days, you’re talking about maybe 8 to 10 hours excluding taxis and flights, not using a checklist, and then making very consequential recommendations on what should be done with the company, things that will impact other people. I have to ask, given all of those constraints, are you just that brilliant, or how exactly do you make those determinations in a single day with no checklist?
Joannes Vermorel: Yeah, I mean, first, the code base. If you’re slightly used to reading a lot of code bases, those things become extremely obvious. Within just a few minutes of looking at the code base, you see if the code is well written or just like crap. I usually have an opinion within 5 minutes of browsing that code base. If you’re really used to reading many programming languages and you’ve seen many code bases, you don’t need hours to forge an opinion on whether this is nice, tight engineering, well-organized, makes sense, or if it’s just pure chaos and inconsistent. For example, if I go through a code base and see radically different coding styles, I say, “Oh, okay, so we have different contributors who have no shared agreement whatsoever on how the code should look like.” It’s really bad. Do I look at the history of commits? That tells me a lot of things. For example, if I see that in the history of commits, you have one commit for a feature and then two dozen commits to fix bugs caused by the introduction of this feature, that’s super bad. That means that whenever people introduce anything, they will then spend a lot of time just fixing the stuff they just broke. So you see, just by having a cursory view at the last maybe 200 commits, you can see if it’s a pattern. You can see many patterns. Sometimes, for example, you have the history of the commit messages. This is the evolution for the audience. Commits are the way you incrementally modify the code base. You make changes, and you typically use Git nowadays. We use that for the web, and at Lokad, we use it like everybody else for our tech stack. The question is, for example, when you see changes applied to the code base, do the changes make sense? For example, if you have commits where the title of the commit is “more stuff,” literally, what is this piece of crap? No, you do not have some stuff that has been changed. Then, for example, if you have a bug that is being fixed, is there anything being done alongside fixing the bug to make sure that the bug isn’t reintroduced next time you touch the code base? There are plenty of ways to make sure that the bug isn’t reintroduced. Are you taking care of that? When you look at contributions, commits, do people have code reviews? Do they discuss how things should be engineered? Again, the code base will tell a huge story both about what the technology is but also its history. You can see the trajectory, how it’s developed. Then you can zoom into the core parts. For example, if you’re an AI or machine learning startup, your core value is supposed to be some smart numerical recipe that does something very difficult or whatever. How are you doing it? Is it just a trick, or do you have some deep expertise? How much moat do you have? Do you have a real moat where replicating what you’ve just done will be difficult, or is it just an illusionist trick where you’re just using some kind of open-source library and bam, actually nothing is your production? You’ve just recycled something that is in the community. Again, that is a very critical question for the investor because if people say, “Oh, we have developed a unique machine learning technology,” if what you’re doing is essentially reusing a piece of open-source, anybody can do it. So yes, it’s cool, yes, it works, but should the investor value your company highly just because you have something that works? Well, not really, because if it’s open-source, it means that pretty much anybody can do it also. So no moat. Again, the things that I’m looking at really depend on the specifics of the company.
Conor Doherty: You can also, even just taking the example and then combining it with a bit of an anecdote, if you take the example of looking at the commit log in Git or Bitbucket—we use Bitbucket—you can extrapolate quite a bit. For example, even for us on the marketing team, whenever we push anything to the website, you insist on a very clean, tidy commit log. You can see exactly what happened, who approved it.
Joannes Vermorel: Exactly. And also, sometimes just the way people interact with me. I’m talking to a person, I ask a random question, is this person lightning fast to just bring me to the right place in the code? Like, “Oh, there is this feature that is kind of interesting, can you show me the code that is relevant to that?” And then the guy says, “Oh yes, no problem,” bam, direct spot on, right into the implementation. That’s very good. That complete mastery of the codebase, of how things are being done, that’s very, very good. So sometimes just the sort of ease that people demonstrate in navigating the codebase tells me a lot. Also, for example, when they introduce features, do they have well-written tickets to justify it? Are the software engineers loose and doing stuff with the code, or do they have something that is well-organized where, “Okay, I want to do that, here is my ticket that cleanly describes why I want to do that,” and then I discuss the different ways to take it, etc. So it depends, but that’s typically the importance of the tickets. It would typically be critical for B2B apps, productivity apps, because you have an endless list of features that are all super domain-specific. If you’re doing an enterprise app, you really should not be getting lost in the infinite amount of features that you could add to your app. So that would be very important for different sorts of companies. That would be a completely different process. But what I’m saying is that in terms of how can you forge yourself—if we go back to that—how can I forge myself an opinion in just a few hours? If you’re just sitting next to someone interacting with the codebase, you can see really a lot. If you’re just paying attention to the patterns, to what people are doing, and yeah, it’s pattern recognition. There’s a finite number of things people can be good or bad at given a certain vocation.
There’s a finite number of things people can be good or bad at given a certain vocation.
Conor Doherty: Yes, and also you check the consistency, you know. Are people even telling the same story in terms of technology? Like, I ask, “What do you think this component is doing?” And if I get several people who give me completely different stories, I say, “Okay, so clearly within this team, it’s not very clear that people really understand what is going on.”
Joannes Vermorel: I don’t need the truth to recognize that things are inconsistent. Again, my goal is not to become an expert in what they’re doing. My goal is just to assess whether the team is functional, the trajectory in terms of development of the technology is sane. Sometimes it’s also about checking, for example, that the resources in terms of cloud computing resources are just under control. Sometimes you have companies who are mismanaging their cloud resources. The cloud is fantastic, pay as you go, which means the sky is the limit. You can spin tons of virtual machines, tons of storage, tons of everything. And sometimes, you know, some companies go a little bit crazy on that and they end up with a trajectory where they are spending way too much on cloud computing resources than they should.
Conor Doherty: That again depends on what they’re doing. How much should you be paying for cloud computing resources? Well, it really depends on what you’re trying to achieve. So there is no clear rule, but part of the experience is to assess what is the workload that you’re trying to accomplish on the cloud and does it make sense that you’re paying that much? Is it an achievement, well done, your software is highly performant, or is it super bad and you’re just burning money? Probably giving you tons of extra money to burn is just not the right move. You should probably be fixing those problems before being granted a few extra million.
Joannes Vermorel: If you’re lacking money, you cannot discipline yourself to not burn money through unwise spendings in computational resources. The chances that you will become wise once you have a lot more money are super thin.
Conor Doherty: On that note, we do actually have another video on the topic of computational resources, so check the playlist for that. But on the topic of money, when you talk about finances, I know when we talk about supply chain, you talk about the bottom line financial impact of whatever choice it is you’re going to make. Do you take the same approach when evaluating a tech company? Do you look at what is the most financially beneficial orchestration of the technological company or what is just the best tech regardless of the price?
Joannes Vermorel: No, no, no. Every startup has limited resources. The idea, and that’s the problem again with checklists, is that checklists have an absolute perspective. They say you need to do this, this, this, and this. My perspective is always completely relative to the maturity of the company.
Conor Doherty: Yes, exactly, on their own terms.
Joannes Vermorel: You don’t want, for example, it would be pure insanity for a tiny company to go for an ISO certification. It’s a complete waste of resources unless it is a hyper-regulated market where you cannot survive unless you have this certification. What I’m looking at is how good are you at making use of your limited resources to get the maximal bang for your buck. I’m just taking as an assumption that your direction is good, that ultimately if you succeed, you’re going to make money. But still, if I take your business directions for granted, I say, “Okay, let’s admit that if this thing is done, then you will be competitive in this market and there will be money to be made.” I take that. Now, every dollar or euro that is being spent on becoming this number one player in this market, is it wisely spent?
Conor Doherty: Absolutely. That can be, do you have the right talent? Sometimes they’re spending too little. It does happen that sometimes the company achieves a degree of success, and they don’t let go of people who were originally hired just because they were super cheap. That also happens quite frequently with tech startups.
Joannes Vermorel: Especially when the founder is not a tech founder, it’s a sales founder, and then you have the CTO who was originally the first software engineer ever hired by the company. This happens frequently, and this person is not at the level. Again, exceptions vary, your mileage may vary.
Conor Doherty: Exactly. My question there is, have there been situations where you have gone into evaluate a company, you are there to evaluate all the things you’ve talked about: the code, the way the team works, could they conceivably accomplish the goal using the resources that they have? But you’ve gone in and their ultimate goal was just nonsense. For example, it’s 2008 and this company is determined to bring back VHS tape, like just something utterly insane that you have a strong opinion is a massive waste of money and time and resources.
Joannes Vermorel: Oh yes, that happens. Feel free to, but don’t give any names.
Conor Doherty: No, the thing is, some software technologies evolve quite quickly. For example, during the last two years, I’ve audited quite a few companies who were doing NLPs. These companies were started in 2018 or 2019, and they went into this AI wave and developed NLP, natural language processing technologies, but pre-LLM, pre-large language models. There was an enormous literature of NLP models and machine learning, deep learning, that is not LLMs. LLMs have been an extinction event for all those NLP technologies.
Joannes Vermorel: Yes, there were many companies where my conclusion was, this company has developed 20 fairly sophisticated NLP models, they are all completely obsolete. LLMs are uniformly outperforming all of that. You can just throw all the tech stack and restart from scratch with an LLM front and center, you will be better. There is nothing to salvage. It happens quite frequently, maybe one out of 10, one out of 20, where the technology is just obsolete for some reason. Sometimes people haven’t seen exactly what is going to be the replacement, but yes, when I see that this thing is just going to be, you are solving a problem that is already solved.
Conor Doherty: Usually, the thing is that it’s not nonsense in the sense that those people are idiots. There is a problem to be solved, and they have developed something that is nice, but there is something much better that has already been found and possibly free. This thing is not yet popular, it’s not super well known yet. I think LLM is really the extreme case where people, those companies were kind of desperate because they knew that LLMs were coming and they were an extinction event. But you’re the founder of a company, you’ve already invested the money that you raised in seed rounds to do that, and now you’re faced with the prospect that your technology is kind of worthless. You try to maybe fetch an investment which would let you redevelop the technology. Difficult.
Joannes Vermorel: My take in this sort of situation is not to say to the investors, “Don’t invest.” My message is, the technology stack that they have is just worth nothing. If the people are good, there might still be a case to invest in them, but just at a much lower valuation because what they have is just, this technological asset is completely out of date. It’s not an asset anymore, it’s a liability.
The team is functional, and the trajectory in terms of development of the technology is insane. Sometimes, it’s also about checking, for example, that the resources in terms of cloud computing resources are just under control. Sometimes, you have companies who are mismanaging their cloud resources. The cloud is fantastic; pay as you go means that the sky is the limit. You can spin tons of virtual machines, tons of storage, tons of everything. Sometimes, companies go a little bit crazy on that and end up with a trajectory where they are spending way too much on cloud computing resources than they should. It depends on what they’re doing. How much should you be paying for cloud computing resources? Well, it really depends on what you’re trying to achieve. There is no clear rule, but part of the experience is to assess what is the workload you’re trying to accomplish on the cloud and does it make sense that you’re paying that much. Is it an achievement well done, your software is highly performant, or is it super bad and you’re just burning money? Probably giving you tons of extra money to burn is just not the right move. You should probably be fixing those problems before being granted a few extra million. If you’re lacking money, you cannot discipline yourself to not burn money through unwise spendings in computational resources. The chances that you will become wise once you have a lot more money are super thin.
Conor Doherty: It occurs to me, and on that note, we do actually have another video on the topic of computational resources, so check the playlist for that. But on the topic of money, when you talk about finances, I know when we talk about supply chain, you talk about the bottom line financial impact of whatever choice it is you’re going to make. Do you take the same approach when evaluating a technological company? Do you look at what is the most financially beneficial orchestration, or what is just the best tech regardless of the price?
Joannes Vermorel: No, no, no. Every startup has limited resources. The problem with checklists is that they have an absolute perspective, saying you need to do this, this, and this. My perspective is always completely relative to the maturity of the company.
Conor Doherty: Yes, exactly, on their own terms.
Joannes Vermorel: You don’t want, for example, it would be pure insanity for a tiny company to go for an ISO certification. It’s a complete waste of resources unless it is a hyper-regulated market where you cannot survive without this certification. What I’m looking at is how good are you at making use of your limited resources to get the maximal bang for your buck. I’m just taking as an assumption that your direction is good, that ultimately if you succeed, you’re going to make money. But still, if I take your business directions for granted, I say, okay, let’s admit that if this thing is done, then you will be competitive in this market and there will be money to be made. I take that. Now, every dollar or euro that is being spent on becoming this number one player in this market, is it wisely spent? Absolutely. That can be, do you have the right talent? Sometimes, they are spending too little. It does happen that sometimes the company achieves a degree of success, and they don’t let go of people who were originally hired just because they were super cheap. That also happens quite frequently with tech startups. Especially when the founder is not a tech founder, it’s a sales founder, and then you have the CTO who was originally the first software engineer ever hired by the company. This happens frequently, and this person is not at the level. Again, exceptions vary, your mileage may vary.
Conor Doherty: Exactly. My question there is, have there been situations where you have gone in to evaluate a company, and their ultimate goal was just nonsense? For example, it’s 2008, and this company is determined to bring back VHS tape. Something utterly insane that you have a strong opinion is a massive waste of money and time and resources.
Joannes Vermorel: Oh yes, that happens.
Conor Doherty: Feel free to share, but don’t give any names.
Joannes Vermorel: Some software technologies evolve quite quickly. For example, during the last two years, I’ve audited quite a few companies doing NLPs. These companies were started in 2018 or 2019, and they went into this AI wave and developed NLP (Natural Language Processing) technologies, but pre-LLM (Large Language Models). There was an enormous literature of NLP models and machine learning, deep learning, but not LLMs. LLMs have been an extinction event for all those NLP technologies. For those companies, my conclusion was that they had developed 20 fairly sophisticated NLP models, but they are all completely obsolete. LLMs are uniformly outperforming all of that. You can just throw away the tech stack and restart from scratch with an LLM front and center. There is nothing to salvage. It happens quite frequently, maybe one out of 10 or one out of 20, where the technology is just obsolete for some reason. Sometimes people haven’t seen exactly what the replacement is going to be. When I see that this thing is just going to be, you are solving a problem that is already solved. Usually, it’s not nonsense in the sense that those people are idiots. There is a problem to be solved, and they have developed something nice, but there is something much better that has already been found and possibly free. This thing is not yet popular, not super well known yet. LLMs are an extreme case where companies were kind of desperate because they knew what was coming, and it was an extinction event. You’re the founder of a company, you’ve already invested the money you raised in seed rounds to do that, and now you’re faced with the prospect that your technology is kind of worthless. You try to maybe fetch an investment which would let you redevelop the technology. Difficult. My take is not to tell the investors not to invest. My message is the technology stack they have is just worth nothing. If the people are good, there might still be a case to invest in them, but just at a much lower valuation because what they have is not an asset anymore, it’s a liability.
Conor Doherty: On the other end of that spectrum, have you ever gone into a company and left thinking, I’ve just met the next Elon Musk? Like, that guy’s a genius, they’re sitting on a diamond.
Joannes Vermorel: On that scale, no. But yes, I did feel sometimes a little bit jealous compared to my own company. Mostly when people have such beautiful niches. There are so many beautiful niches. The cases that make me more jealous are beautiful problems that are super niche. Nobody even knows that this problem exists. It turns out there are tens of millions, possibly a billion at stake. This is a niche market, and some people bring a solution for this niche, and it’s beautiful. It’s very simple, straightforward, to the point. Instead of going through a hyper-competitive market, for example, Lokad doing supply chain optimization, we have over a thousand competitors worldwide. It’s a very crowded market. We have some competitors that are multi-billion companies. The competitive landscape is tough. Sometimes I meet companies where they literally have zero competitors. They are the one. Their clients are happy, they have a solution that is really tuned to a niche problem that I have never heard of before.
Conor Doherty: Examples there or would that be to reveal too much?
Joannes Vermorel: No, that would be unfortunately too revealing. I try to, but the bottom line is, think of, let’s say, anything that is super specialized.
Think of specialized maintenance of an offshore oil rig. There are tons of things that need to be done, and some special operations need software as a piece of support. There are companies that do that. Think of every single niche that probably needs some very dedicated solutions.
You have plenty of situations where you can imagine how much technological equipment a surgeon uses in an operating room. Tons of equipment, even if you want to measure something. For example, in this building, they had a control to detect whether we had asbestos. There was a piece of equipment with software to detect asbestos in this building. There is a company somewhere doing that. These can be very specific niches that may look small, but when you look at the global market, it’s not that small.
These super small niches might be a few hundred million dollars worldwide, and a company has the opportunity to take 50% market share because there are no competitors. The alternative is people using traditional pen and paper methods.
Conor Doherty: Over the course of those 10 years and having seen a wide array of different examples, are there any general best practices and worst practices that you’ve noticed in terms of software or startups? Let’s start with the things to avoid and end on a positive note. So, things to absolutely avoid?
Joannes Vermorel: If you do not care about your technology, it’s going to blow up in your face. Lack of care is a killer. Whenever I see companies where people do not care about the technology, usually the technology is complete crap.
Conor Doherty: Define what you mean by “don’t care.” Is it that they’re not interested in learning about it, or they did at one point and now they don’t, or is it a lack of skill? What do you mean by that?
Joannes Vermorel: The best way to describe it is, when the CTO takes a shower, does he think about his technology or does he think about golf? People who genuinely care have a sort of intensity. It doesn’t stop at office hours; it goes beyond that. They are extremely curious, eager, and passionate. They want to do the right thing truly. There is this element of slight insanity where it’s not just about doing what is right for the client, but also doing what is right for the technology itself.
Are you engineering something that is truly beautiful? There was this anecdote about Steve Jobs. He wanted the iPhone to look good even on the inside. When you tear it apart, the components inside the iPhone would be elegant themselves, with the right choice of colors and things. This degree of care reflects that you care deeply, not just for how it looks on the surface, but also for the technological parts that no customer will ever see.
When people do not care, it is really bad. The worst is probably when tech companies outsource the development of their technologies to freelance contributors. It doesn’t matter how good or bad those freelancers might be; the net result is that the technology is usually a complete mess.
Conor Doherty: I think most people would agree that inherent passion and drive to do a thing well is a fantastic quality. That said, if you only have 20 minutes with the CTO, how do you gauge those intangibles without a checklist?
Joannes Vermorel: Someone with intensity will tell me, “We took this technology because of this reason.” They know the why. It’s not just, “I needed a web interface, so I picked this one.” They have opinions on their technological choices. People with passion have so many opinions; it’s leaking out of them. They can’t help themselves. Every time I touch a part of the technology, they say, “We do it this way because of these reasons.”
Conor Doherty: It’s the passion that they’re interested in.
Joannes Vermorel: Yes, and you can see the intensity of their thinking. It’s not just a piece of technology where they did it because there were requirements. People who produce code by the kilometer have a super pedestrian approach where requirements come from the sales team, and they just implement them. That is a recipe for a tech stack that is super crap.
Conor Doherty: For example, if you had audited Steve Jobs in the past and he told you, “Look at the inside of this phone, I’ve made it beautiful,” your memorandum would say he’s passionate, but that’s a crazy thing to spend money on.
Joannes Vermorel: No, no. The question is, does it feel like they are going for distractions, or do they just do the extra polish? Those things may not be very costly. It’s just the intensity. You do a few percent more so that you have the thing really tight and nice. That’s not insanity. People who overdo it typically have a second system effect. It’s when a team of engineers, in their second attempt, over-invest in things they don’t need because the first time failed due to crappy tech.
The most frequent case is when people over-invest in scalability that they won’t need anytime soon. They say, “Look at our beautiful architecture; it can deal with 10 million users.” How many users do you have at this point? “Oh, 200.” There’s a vast disconnect between what is needed and the over-engineering.
Also, the people who are patient about the wrong things. Do you have a patience that is for a certain elegance and beauty of technology for itself, or are you going through some kind of dogma? In software industries, you have all sorts of dogmas floating around. We need to do it like Scrum, or we need to do domain-driven development, or test-driven development, or microservices. There are plenty of guru-esque takes in the software industry, and you have people who are zealots for this. When I say intensity, I mean people who love what they are crafting but who are not zealots about a specific ideology. Zealots do things that completely mismatch even their own practical requirements. They say, “Oh, but we have to do it that way because it’s test-driven design.” The fact that it’s test-driven design is not a justification for doing it that way. Does it really make sense to do it that way with regard to whatever problems you have?
Conor Doherty: And in terms of the best practices, don’t just say the opposite of what you just said. Are there any off the top of your head things that, when you see them, you think, “Oh, that’s a good sign”?
Joannes Vermorel: Clear writing. You said that before. Clear writing. Every single piece needs to be to the point, sharp, and concise. Do you have pages of comments in the code that are just bureaucratic, like, “Oh, we need to comment this”? Is there a template, and you have a page of comments nobody is going to read? Or do you have super tight comments that are very interesting? Just by reading one comment, it sheds light on the entire code file. When I look at the tickets, is your problem statement written in a way that is very clear, or is it an incomprehensible mess? When you have a technical challenge, is there a very articulate discussion of the problem and the possibilities, or do you have a crappy slide with insane bullet points that say nothing? Quality of writing is one of the telltale signs. Another one is when people use unusual technologies that still happen to be very good choices. If they use an unusual piece of technology, very frequently it is just out of ignorance. They didn’t know there was a much more mainstream solution that would be much better. Usually, it’s a negative, but when it turns out that a relatively rare piece of tech is surprisingly to the point for what they’re doing, that’s very good. It means they have explored the technological landscape, especially the open-source technological landscape, in a very extensive way. That’s very nice. Nowadays, nobody would be surprised if people say, “Oh, we’re using Python, PostgreSQL, Tailwind for CSS, React.” Those are components every halfway competent software engineer knows about. But if they know something super specific that really fits their problem, that is also very interesting.
Conor Doherty: Cool. Well, we’ve been going for about an hour. I’m mindful of time, so to wrap things up, are there any closing thoughts you’d like to share with everybody?
Joannes Vermorel: Yes. I think what surprises me the most in this decade of audits is how frequently tech companies can literally spend years and sometimes successive rounds of investment with nobody having a look at the tech. You would think that Theranos, this big fraud, was an exception. Such a massive investment, and nobody did any due diligence of the technology. But the funny thing is that it’s the norm. For me, what is very intriguing is how rare these technological audits are. When I talk with those startup founders, they usually tell me, “We had no idea you would be asking those sorts of questions.” They think I will come with this insane, stupid checklist. Just imagine Theranos; people would have easily fooled the checklist. There were plenty of checklists around, like, “Are you compliant with this and that?” Yes, yes, yes, yes. But the core question, which was, “Is your blood analysis technology real?” Nobody asked those questions. They were super basic.
Conor Doherty: This comes back to the point you made before about skin in the game. If venture capitalists are going to invest millions in your company, they have skin in the game.
Joannes Vermorel: The thing I find very surprising is that usually, even if I’m coming in the third round, nobody has had a serious look at the technological stack. It’s very strange. You have auditors who are just ticking boxes, but in my book, those do not count. They just pretend to do the work. They don’t do anything of substance. It is super easy to fool those people because all you have to do is lie. Just say, “Do you do that?” “Yeah, we do it.” “Show us.” You show something that doesn’t make any sense. The auditor is not a tech person, so you could show him anything, and he would just say, “Okay, box ticked.” At the end of those audits, it’s very intriguing. In classical audits, the auditor will say, “Please sign this paper that says you didn’t lie to us, and if you lied, we decline all responsibility.” My take is that I’m certainly not going to do that. I do not assume that people are telling me the truth. That’s why I want to see the codebase. Yes, you can fool me and tell me tons of things, but when we look at the codebase, forging a codebase, forging thousands of commits, forging hundreds of tickets, forging all the supporting documents, that is way more difficult. What surprises me is that the cost is one day of audits. Yes, I’m not exactly cheap, but still, it’s one day. Even after doing these audits for more than a decade, I still notice that when I come, even in the second or third rounds, nobody has really challenged the tech. People just assume it’s good, it works, they will do their roadmap. No check whatsoever. For me, that’s a little bit baffling.
Conor Doherty: Well, hopefully, something we’ve discussed today will make a difference, perhaps even inspire a few people to reach out to you. You’re also available for bat mitzvahs and birthdays, I believe.
Joannes Vermorel: Yes.
Conor Doherty: Well, Joannes, thank you very much for your time and for answering all my questions. Thank you very much, and thank you for watching. We’ll see you next time.
