Artificial Intelligence in IT: pitfalls and limitations

Currently, tech companies such as SAP consider AI to be a core part of their business strategy, increasingly embedding AI systems into their existing solutions (e.g. SAC predictive planning or the PAL and APL libraries in SAP HANA) or developing new AI solutions altogether (Joule). This is due to the fact that organizations are embracing the value of AI being integrated into their IT landscape, enabling them to make informed business decisions and optimize their business processes.

However, the adoption of AI solutions in our society is a double-edged sword. While a successfully deployed AI solution (e.g. OpenAI’s ChatGPT) can give companies an advantageous position in this fast-paced market, a wrongly implemented AI solution could inhibit the productivity of an organization, potentially becoming a costly mistake in terms of its revenue and reputation. In one example, British delivery service DPD recently implemented chatbots to handle customer service tasks. However, DPD was forced to shut them down when it was shown that the chatbots could be influenced to generate a poem harshly criticizing the organization and publicly embarrassing them. In another more serious example, online real estate marketplace Zillow lost around 420 million dollars in the third quarter of 2021 due to a machine learning algorithm failing to predict home values correctly. This forced Zillow to shut down its division responsible for acquiring and selling homes, firing twenty-five per cent of its workforce.

These examples illustrate that AI systems have their limitations and that releasing such systems has its pitfalls. With this in mind, what exactly are the pitfalls and limitations of AI?

One of the most common pitfalls seen when developing AI solutions is that machine learning models will automatically learn valuable relationships from their training data as long as there are enough entries in a dataset. However, machine learning models only estimate relationships and biases that are already present in the training data. Particularly, if the training data is not of high quality, then AI models' output will also be flawed. Conversely, building an AI model on high-quality data will yield better results. To illustrate, if no proper data cleaning has been performed on the training data, the AI model will most likely produce incorrect results by learning relationships that do not make sense in the context of the business case. This is why data cleaning is one of the most important stages in any AI project, as it prepares the machine learning model to learn the optimal and correct relationships from the data. Thus, the ‘power’ of AI solutions lies in ensuring the quality of data volumes, not necessarily in their complexity.

Another common pitfall seen in many AI projects is that the focus shifts to developing the most accurate model, assuming that the more accurate the model, the more reliable its results. In this case, the solution to the business case is equated to the values of some quantitative metrics measuring the model’s accuracy. While an accurate model is important within an AI solution, even the most accurate model can make serious mistakes. As such, the output of AI models should not be taken at face value when making important business decisions. Instead, they should be considered as indications that serve as a discussion point in determining the optimal business strategy. Ultimately, the results of an AI solution should always be evaluated in terms of the impact they have on the organization’s value (e.g. the amount of revenue lost in case of mistakes produced by the AI solution).

There is more to realizing an AI solution than simply collecting large amounts of data and training a machine learning model on it. Each step in the development and maintenance of an AI solution should be carefully evaluated in terms of the value it generates for the corresponding business case. As such, releasing AI solutions into the market should be a gradual and incremental process. However, the great diversity of AI models and data processing techniques makes it difficult to envision the best possible solution for the business case, as distinguishing among the different methods usually requires domain knowledge. Nevertheless, there are some tips and tricks that can guide you in selecting the optimal methods within an AI project. Stay tuned for the final part of this blog series, where we will explore some of these best practices.

Have you also read part 1. of my blog on AI? You can read it here.