Is This the Right Team?: Inconsistencies as a Yellow Flag during Diligence

I imagine you see start-ups with similar ideas multiple times a day! As you know, at the earliest stages, the focus of diligence is often about figuring out if “this is the right team.” You've already confirmed that the team has the necessary expertise and has done this before. Here’s what else to look for.

Inconsistencies, which come in many different flavors, come up a lot when I diligence start-ups – and may be an early indicator that this may not be the right team to execute the idea that will return the fund. Here’s why that is, what to look for, and what to ask about in diligence.

Algorithmic Inconsistency

As someone who does a lot of product testing as part of (AI) diligence, one of the (many) things I look for, besides how it solves the customer pain problem end-to-end, is how consistent the results are.

I imagine you’ve already evaluated the product recommendations and found them to be accurate, precise, and reproducible. (And I don’t mean that the algorithm reached 95% accuracy, which can be a misleading metric in its own right, and one I’ve previously written about here.)

Do you currently verify the consistency of the outputs and/or any resulting recommendations that are a part of the product offering? That is, does the order of the inputs matter to the outputs? And more importantly, do the recommendations or insights that the customer receives change when the input order changes? Most start-ups overlook this crucial step, which makes the product not trustworthy!

One way this shows up in product testing is, for example, say we're comparing test scores of males and females before deciding how to support the underperforming group better. If the algorithm implementation were not consistent and everything else stayed the same, except the order of the inputs, we would see different results had we compared males to females versus females to males! I’ve seen this happen earlier this month; I wish I were joking!

This start-up wasn’t even using an LLM (or a tool such as Cursor or similar, which are LLMs under the hood) to make recommendations (!), where inconsistency is a known issue and an ongoing area of research. If you’re evaluating a start-up that’s using an LLM or a tool that’s using an LLM under the hood, now you know it’s even more important to check for this!

Algorithmic inconsistency suggests that the team is lacking expertise in AI; if AI is core to the business, then this expertise should be a part of the team (even in advisory capacity) from the beginning, if nothing else, then to help manage expectations about the technology between the team, investors, and customers.

Workflow Inconsistency

Workflow inconsistency comes up a lot, especially in HealthTech start-ups, when the founder(s) don’t have a healthcare background and have no advisors (or anyone else) on the team to guide them. While I also want healthcare to work as these start-ups envision, that’s typically not the case.

If the team lacks industry experts now, how will they hire, synthesize, and prioritize customer requests to deliver the most value and company growth?

Strategy Inconsistency

When I’m not evaluating start-ups, I mentor and collaborate with over 100 start-ups a year! One of my favorite questions to ask co-founders and executive teams is: What are the top three priorities for the company over the next six months? Three years? The majority are not on the same page! (I share advice on this topic, as well as on how to “Build and Scale with AI,” in an invited talk by Slauson & Co.; you can find it here.)

If the team is not on the same page now, with only a few collaborators, how will they align and prioritize as the company grows?

Implementation Inconsistency

I’m always disappointed when I come across a significant gap between what founders pitched and what actually exists in the product. This implementation inconsistency fosters distrust on all sides.

If this is how the team is also selling to prospective customers, the churn rate will be high, as there will also be a lot of disappointment! Given that investors are often likened to marital partners, as relationships typically last more than 5 years, is misrepresentation (even inadvertently) the right approach to starting a prospective relationship?

I’ve also seen this inconsistency many times in start-ups pitching recommendation systems, but then all the product does is return “12”... (I wish I were joking!) And it’s not clear: where did this number come from, why is it 12 and not 15; how can we make it 10 – and should we?

Implementation inconsistency suggests that the team is lacking expertise in the customer’s industry.

Technical Inconsistency

Technical inconsistency arises when founders buy into the technical/AI hype without understanding the underlying mechanics, or when there is miscommunication between non-technical founders and their technical collaborators.

I can’t tell you how many start-ups pitch that their product, an LLM wrapper, either works out of the box in a niche industry that the 3rd-party LLM has not actually trained on (but the team hopefully added a few examples (or few-shot prompts) to the LLM), or claim that it will return exact answers in industries where there is no room for error, when that’s not how LLMs operate. (I expand on the latter topic in this blog post to illustrate just how much LLMs misunderstand the IRS Form 1040, where exact answers are required.)

Many non-technical founders struggle to communicate with their technical counterparts! One solo non-technical founder building an AI-native product proactively offered to share their architectural diagram during diligence to help everyone better understand the stack. But they shared their business diagram! (I have nothing against solo founders, and often suggest that founders don’t need co-founders!) I would argue that the diagram of how the business is/should function is more important, but that’s not what they promised! Since they’re non-technical, it’s a potential yellow flag that they don’t have the right team in place to guide them. The miscommunication that’s already happening between the technical and non-technical collaborators is more likely to compound as the product evolves!

Advice for Diligence