Mislav Jurić

Category: AI in business

Posts pertaining to AI in business.

  • Three ways I’ve watched AI projects fail (and how to avoid them)

    Introduction

    As I talked about in my previous blog post, the failure rate of AI projects is between 80% to 95%. In this post I will give my take on why this happens after having worked on various AI projects for 5+ years, both as an engineer and as a consultant. This post will be filled with “war stories” of projects I worked on and the lessons I learned from them.

    Do note that I will most likely not enumerate all of the failure modes of AI projects; I will enumerate only the ones I’ve seen on the projects I worked on. Let’s get started.

    There is an existing non-AI solution to the same problem

    This failure mode happens when businesses try to solve a problem with AI where there already exist non-AI solutions which work well. In particular:

    I worked on a project in a traditional industry. The project aimed to use AI to solve a problem which already had a non-AI solution, but the premise was that the AI solution would be better. I was the lead developer on the project: I took it from concept to deployment and had regular meetings with the client to make sure we are aligned on the project development direction.

    As I learned more and more about the industry, I realized that the entire premise of the project was false. The project claimed to solve a problem which had already been solved by non-AI methods in a better way, but I found that it would be slightly faster at best.

    I am no longer actively involved in the project, but I hear some news about it here and there. Essentially, the product (which was the result of the project) is still searching for its customers. This makes sense: customers want tried-and-true solutions to their problems and the only differentiator between this product and others is that it is slightly faster and uses AI. There really is no significant value-add other than that. I think that fundamentally customers recognize that and that’s why this product is still struggling to get customers to this day.

    Also, you have to take into the account that since this is a traditional industry, customers are generally more skeptical of new technology over tried-and-true methods, so unless there is a really significant value add to justify the risk of adopting new technology, most customers won’t convert.

    How to avoid this type of failure

    If you will be creating an AI-based product, make sure that it outperforms existing solutions (especially in traditional industries). Otherwise, you end up with a product which is competing with tried-and-true non-AI based solutions and there’s no strong reason to prefer your AI-based product over traditional ones.

    Unrealistic expectations of what AI can do

    This failure mode happens when the expectations of what AI can do are unrealistic. Read on below:

    I was working on a project in the entertainment industry. The goal of the project was to enable immersive experiences by using AI as a central component of the product.

    As we worked on the product, we realized that AI capability has its limits. Even with state-of-the-art AI models, we couldn’t achieve some of the things which were part of the product vision. And we really had a team of brilliant engineers building this product, but we just couldn’t enable some things as technology wasn’t (and even today isn’t) advanced enough.

    We had two choices at one point during project development:

    • Continue using AI as a central component of the product or
    • Switch to using non-AI methods for the majority of the product and use AI where it shines.

    The decision was made to continue using AI as a central component of the product and to make AI match the original project vision. At some point during its development, the project was released into the public to users, but the economics weren’t working out; there was not enough users to justify the cost of running the team building the project. After some time, the project was restructured and all of the team working on the project was let go (including myself).

    How to avoid this type of failure

    When building AI-based products, test your core AI assumption early: can the technology actually do what you need today, or are you betting on future improvements? If you’re choosing between doubling down on AI or pivoting to hybrid approaches, bias toward the pivot. Doing research and development to advance AI capabilities which the AI-based product will rely on is not a good strategy.

    AI project is poorly managed

    This failure mode most often manifests itself in unrealistic deadlines and constant firefighting. Read on more below:

    I was hired to provide leadership guidance to an AI-based retail analytics project. Immediately as I joined, I was informed about impending deadlines. After taking a look at the situation, I explained to the leadership that the scope of what I can do until the next deadline is very limited.

    As time went on, it became clear that the deadlines which were set by the management were unrealistic. Once I realized that, I spent some part of my time explaining to the management that what they want to build will take longer (much longer) than they initially expected. That was a hard pill to swallow, but ultimately they appreciated me for my honesty and for setting the project in the right direction.

    How to avoid this type of failure

    When planning AI projects, it pays to involve machine learning engineers in the process. They will be able to provide good ballpark estimates of how long something will take. Ideally, it’s best if the person doing the project planning also has some engineering experience, because then their estimates tend to be more accurate, but the next best option is to have a trusted engineer who can give you a realistic estimate of how long something takes.

    Conclusion

    I hope that this blog post has given you some clarity on why AI projects fail. As I said in the Introduction, this isn’t an exhaustive list, but rather it’s a list based on my own experience from failed AI projects. These patterns show up repeatedly in AI projects. Recognizing them early – before you’ve committed significant resources – can save you from joining the 80-95% failure rate.

    As a machine learning engineer and consultant, I do my best to help businesses avoid failure while developing AI projects (among other things). If that sounds like something you could use, feel free to reach out to me.

  • Does your business need AI?

    Introduction

    Almost everyone is talking about AI today. With the advent of LLMs and their continuous improvement, the field of AI (machine learning) has grown in popularity in recent times. With it, a lot of business owners are wondering whether or not their business needs AI.

    In this blog post, my aim is to honestly tell you whether or not you need AI. I will base this on my experience (both as a consultant and an engineer). It is intended primarily for business owners. I believe that today a lot of business owners are being told they need AI, but most don’t know how to evaluate if it’s right for them. The aim of this post is to help you with this decision.

    This blog post is structured in 3 parts. In the first part, I will describe some signs which indicate that your business probably doesn’t need AI. Then, I will explain what you can realistically expect from committing to an AI project. Finally, I will describe signs that suggest you could actually use AI. Let’s begin.

    Signs you probably don’t need AI

    Your business is meeting (or exceeding) business goals

    If your business is meeting (or exceeding) business goals, you probably don’t need AI. Notice the wording here: I said need, not want. You could still want to use AI within your business, but you don’t need it because even without AI, you’re still meeting your business goals.

    It really is that simple: If you are already meeting (or exceeding) your business goals without AI, you don’t need AI. You could experiment with AI in order to try to make some business processes more efficient, but this is not a necessity; it’s more of a nice-to-have. If you are considering using AI even though you don’t need it, I encourage you to read AI realities section below. It’s a sobering view of the realities of most AI projects.

    You want to implement AI because you heard it’s the next big thing or because your competitors are doing it

    Another signal you probably don’t need AI is if the reason you want AI is because you heard it’s the next big thing or because your competitors are doing it. The main reason I say this is that most AI projects fail (around 80-95% of them). I will expand on this in more detail in the AI realities section below, but the point is that most AI projects fail.

    If you still want to use AI, ask yourself this: “If I don’t need AI, but still decided to pursue an AI project and it failed, would I regret it?”. Statistically, this is the most likely outcome. As an AI consultant and engineer my goal is to help businesses avoid project failure, but sometimes this means telling business owners that I wouldn’t recommend AI for their business.

    Takeaway from this section

    If you identified either as someone whose business is already achieving its goals or as someone who wants to try out AI because it’s the hot new thing and everyone else is doing it, I would encourage you to rethink your decision to fund an AI development project for your business. Next section will provide a sobering view of AI development realities.

    AI development realities

    Costs

    AI projects typically require investments ranging from tens of thousands to hundreds of thousands of euros, depending on scope and complexity. More importantly, many companies fail to account for ongoing maintenance costs. In my consulting work, I met with clients who think that once you set up AI, it’s done once and for all. Not quite. Just like non-AI software requires maintenance and regular updating, so does AI. In fact, with AI it’s even more complicated.

    To give you an example: Suppose you built an AI system which analyzes camera feed from banks and looks for threats. After a while, you get an idea to deploy this system to monitor ATMs (which can be outside). This presents a problem: the indoor video recording conditions most likely don’t match the outdoor video recording conditions. This would most certainly impact AI performance in a negative way. What you’d have to do is either train a new AI model for the outdoor environments specifically or retrain the existing one to work good on both indoor and outdoor video feeds.

    A lot of the times it happens that AI system performance degrades even though you use it for the same task. This can happen if there’s a mismatch between the data which was used for AI model development and the data coming into the AI model when it’s deployed in production. In this case, you have to update the AI model in order to improve its performance.

    Time to value can be months to years

    The reality of most AI projects is that it takes months (or even years) until you see a ROI. The reason for this is because it takes at least one month to develop a proof-of-concept (usually multiple months) which validates the idea of using AI at all. Then it takes months until you deploy that AI system in production and make sure that it’s stable enough.

    If you don’t have any data which can be used for training the AI (which we will discuss below), you can expect your timeline to be extended by several months. Also, if you want to fine-tune existing AI models that will add weeks (or months) to your timeline.

    AI projects have high failure rates

    According to this report from MIT, 95% of GenAI projects fail. For reference, LLMs fall under the GenAI category. Some other estimates estimate 80% AI project failure rate. Based on my experience, I would say roughly 70-80% of AI projects fail.

    Why does such a large percentage of AI projects fail? I think this warrants another blog post, but the point remains: about 4 out of 5 AI projects fail. If you are uncomfortable with this statistic, I would urge you to reconsider your decision to pursue an AI project. I am sure that all of the failed AI projects were sure they would beat the statistic, but ended up being the statistic.

    You need to have data that AI could learn from (or collect it)

    In order to develop an AI project, in the vast majority of cases, you need data. If you already have it, that’s great; if not, you will have to collect the data for AI model development.

    This is not as straightforward as it sounds for some projects. For example, you could face issues with GDPR when collecting the data. For other projects, there’s other laws that apply, like HIPAA, where all protected health information (PHI) must be handled appropriately. All of this can either outright disable you from pursuing AI development or make it more expensive and time-consuming to collect the data, which is a prerequisite to AI model development.

    Takeaway from this section

    As we have learned in this section, AI development is:

    • Expensive,
    • It takes long to see value from AI,
    • The failure rate of AI projects is high and
    • You need to have data that AI can learn from.

    This is why pursuing AI development is an “expensive sport”. In my opinion, it should only be done if you have exhausted other options and can’t meet your business goals without AI.

    One additional thing I wanted to emphasize here: You can think of AI projects in 2 phases: first phase is research and development where you develop proof-of-concept and you prove (or disprove) that AI actually helps in that particular scenario and the second phase is putting the AI model in production. If you think about it in this way, it makes sense that the failure rate is as high as it is because all research projects are uncertain. I’ve worked on multiple different AI projects and every one of them was unique. Some of them used more tried-and-true AI methods, while some used state-of-the-art methods, but in reality you cannot really know in advance whether or not AI will work for a project until you test it out.

    Now that we’ve discussed both signs which signal that you probably don’t need AI and took a realistic look at what AI development looks like, let’s turn our attention to the signs that you could actually use AI.

    Signs you could actually use AI

    You can articulate a clear, measurable business outcome (not just “use AI”)

    If you don’t have a clear and measurable business outcome from implementing AI in your business, you probably shouldn’t develop AI.

    Why? Because not having a measurable business outcome leads to vagueness. Did AI help? Did it not help? It’s OK if something to be measured is subjective because not all things can be objectively measured or quantified, but it’s important to agree on how you’ll measure success before you begin development. Otherwise, no one can really say whether AI helped or not.

    On the other hand, if you can articulate a clear, measurable business outcome you have the prerequisite foundation for implementing AI. Then you can measure your metric of interest prior to AI and post AI and compare the results.

    You have a problem that’s expensive to solve using traditional methods (humans can’t process the volume, traditional software doesn’t work)

    If you have a problem that’s expensive to solve using traditional methods, it’s a sign that you could actually use AI.

    To give you a concrete example that I worked on: Suppose you want to track user engagement metrics inside a store (number of unique customers, how long they engage with certain products etc.). Suppose this store has multiple cameras and you want to track these metrics across all cameras inside the store.

    Let’s say you hired a human to do it. How effective would a human be at tracking unique people across multiple cameras? My guess: not much. You can only really focus on one person at a time.

    Now suppose you hired more people (2, 3, 10+). Would they be more effective at the task? Probably not. Even if each person looked at one camera feed, they couldn’t know how many unique customers were in the store; they’d have to synchronize in real time, which is not really feasible. Therefore, in this problem humans can’t process the volume, so hiring more people isn’t a viable option.

    Now assume that you wanted to write traditional software for this task. Spoiler alert: It wouldn’t work. A rule of thumb: Whenever you are dealing with videos or images, unstructured text or predictions, you most likely need AI. In this case, traditional non-AI software wouldn’t cut it and you need AI.

    Now let me give you an example where you don’t need AI: Suppose you have an increased number of customer support tickets and you are thinking about AI. Here AI is not really necessary; you could hire additional people to work in your customer support team to handle the increased volume. This is another example of a situation where you don’t need AI. You could still want it, but given everything we discussed in the AI realities section, I hope that you would at least consider hiring someone instead of going head-first into AI.

    The cost of errors is acceptable

    Another important point: If the cost of errors is acceptable, AI is more likely a good fit than not.

    Why? Because AI systems are non-deterministic, which is a fancy way of saying that they can produce different outputs for the same input.

    Imagine that you were flying on a plane powered by AI and the pilot said: “In 99 out of 100 flights, all is well. However, on 1 out of 100 flights the AI system outputs the wrong parameters and we crash.” Would you fly? I most certainly would not.

    That is why if you’re dealing with mission-critical systems, you should carefully consider whether or not AI makes sense. I worked on multiple AI healthcare projects so far, but these projects have stricter regulations than projects in other industries.

    Rule of thumb: If a mistake in the AI system won’t cause any big damage (people dying, infrastructure going down etc.), AI could be a good fit.

    Takeaway from this section

    If you have:

    • A clear, measurable business goal,
    • You can’t solve your problem by hiring more people or traditional software and
    • The cost of errors is acceptable

    Those are the signs which indicate that you could actually use AI.

    By the way, notice how I wrote “Signs you could actually use AI” as the title of this section; I didn’t write “Signs that you need to use AI”. This is because none of these signs point to the conclusion that you absolutely need AI. However, they are indicators that you could use AI, especially if you tried hiring additional people and traditional software and it didn’t work; that’s a very strong indicator you could use AI.

    Conclusion

    I hope that this post has given you some clarity on whether or not you need AI in your business and that it has given you a realistic perspective on the realities of AI projects. In my opinion, most businesses don’t need AI. I believe that in this GenAI wave (which includes LLMs like ChatGPT) a lot of businesses will lose money on vaguely defined projects which end up not delivering business value and ultimately end up as a number in the AI projects failure rate statistic.

    As a machine learning engineer and consultant, I help businesses figure out whether or not AI makes sense for them. If it does, I help them understand what AI development entails and I try to build a proof-of-concept as fast as possible, so together we can see how AI behaves in their particular use case. If the results of the proof-of-concept are satisfying, I can take the model to production. If you need help figuring out whether or not AI makes sense for your business, reach out to me.