Introduction
As I talked about in my previous blog post, the failure rate of AI projects is between 80% to 95%. In this post I will give my take on why this happens after having worked on various AI projects for 5+ years, both as an engineer and as a consultant. This post will be filled with “war stories” of projects I worked on and the lessons I learned from them.
Do note that I will most likely not enumerate all of the failure modes of AI projects; I will enumerate only the ones I’ve seen on the projects I worked on. Let’s get started.
There is an existing non-AI solution to the same problem
This failure mode happens when businesses try to solve a problem with AI where there already exist non-AI solutions which work well. In particular:
I worked on a project in a traditional industry. The project aimed to use AI to solve a problem which already had a non-AI solution, but the premise was that the AI solution would be better. I was the lead developer on the project: I took it from concept to deployment and had regular meetings with the client to make sure we are aligned on the project development direction.
As I learned more and more about the industry, I realized that the entire premise of the project was false. The project claimed to solve a problem which had already been solved by non-AI methods in a better way, but I found that it would be slightly faster at best.
I am no longer actively involved in the project, but I hear some news about it here and there. Essentially, the product (which was the result of the project) is still searching for its customers. This makes sense: customers want tried-and-true solutions to their problems and the only differentiator between this product and others is that it is slightly faster and uses AI. There really is no significant value-add other than that. I think that fundamentally customers recognize that and that’s why this product is still struggling to get customers to this day.
Also, you have to take into the account that since this is a traditional industry, customers are generally more skeptical of new technology over tried-and-true methods, so unless there is a really significant value add to justify the risk of adopting new technology, most customers won’t convert.
How to avoid this type of failure
If you will be creating an AI-based product, make sure that it outperforms existing solutions (especially in traditional industries). Otherwise, you end up with a product which is competing with tried-and-true non-AI based solutions and there’s no strong reason to prefer your AI-based product over traditional ones.
Unrealistic expectations of what AI can do
This failure mode happens when the expectations of what AI can do are unrealistic. Read on below:
I was working on a project in the entertainment industry. The goal of the project was to enable immersive experiences by using AI as a central component of the product.
As we worked on the product, we realized that AI capability has its limits. Even with state-of-the-art AI models, we couldn’t achieve some of the things which were part of the product vision. And we really had a team of brilliant engineers building this product, but we just couldn’t enable some things as technology wasn’t (and even today isn’t) advanced enough.
We had two choices at one point during project development:
- Continue using AI as a central component of the product or
- Switch to using non-AI methods for the majority of the product and use AI where it shines.
The decision was made to continue using AI as a central component of the product and to make AI match the original project vision. At some point during its development, the project was released into the public to users, but the economics weren’t working out; there was not enough users to justify the cost of running the team building the project. After some time, the project was restructured and all of the team working on the project was let go (including myself).
How to avoid this type of failure
When building AI-based products, test your core AI assumption early: can the technology actually do what you need today, or are you betting on future improvements? If you’re choosing between doubling down on AI or pivoting to hybrid approaches, bias toward the pivot. Doing research and development to advance AI capabilities which the AI-based product will rely on is not a good strategy.
AI project is poorly managed
This failure mode most often manifests itself in unrealistic deadlines and constant firefighting. Read on more below:
I was hired to provide leadership guidance to an AI-based retail analytics project. Immediately as I joined, I was informed about impending deadlines. After taking a look at the situation, I explained to the leadership that the scope of what I can do until the next deadline is very limited.
As time went on, it became clear that the deadlines which were set by the management were unrealistic. Once I realized that, I spent some part of my time explaining to the management that what they want to build will take longer (much longer) than they initially expected. That was a hard pill to swallow, but ultimately they appreciated me for my honesty and for setting the project in the right direction.
How to avoid this type of failure
When planning AI projects, it pays to involve machine learning engineers in the process. They will be able to provide good ballpark estimates of how long something will take. Ideally, it’s best if the person doing the project planning also has some engineering experience, because then their estimates tend to be more accurate, but the next best option is to have a trusted engineer who can give you a realistic estimate of how long something takes.
Conclusion
I hope that this blog post has given you some clarity on why AI projects fail. As I said in the Introduction, this isn’t an exhaustive list, but rather it’s a list based on my own experience from failed AI projects. These patterns show up repeatedly in AI projects. Recognizing them early – before you’ve committed significant resources – can save you from joining the 80-95% failure rate.
As a machine learning engineer and consultant, I do my best to help businesses avoid failure while developing AI projects (among other things). If that sounds like something you could use, feel free to reach out to me.
Leave a Reply