The mechanics of leveraging AI and ML to solve problems

Take a look around. Cars are driving themselves. Automation is everywhere. Even drones have become household toys.

These are clear signs that artificial intelligence (AI) and machine learning (ML) have become democratized.

As powerful as they are, using AI and ML to solve business problems is not a simple process. Many project teams do not fully understand how to work with these technologies.

Algorithm libraries are the smart choice 

There are more than 50 common ML algorithms, all with different applications.  It can be tempting for AI/ML adopters to develop their own in-house solutions. However, building new ML algorithms can be a huge challenge. Significant investments are required and it is difficult to compete with established players such as Amazon (AWS), Microsoft (Azure), and Google. A better alternative is to hire engineers who can work with existing ML libraries built into major cloud platforms.

Off-the-shelf ML algorithms are best suited for evolving markets that need rapid access to working solutions. Such pre-existing, bug-free algorithms are an important advantage as they speed up the development and deployment process.

Not all business problems can be solved using algorithm libraries. More complex problems will still need to be built from scratch by experienced R&D experts. This type of work requires a lot of creativity and human ingenuity. 

Solving problems with AI and ML

AI/ML projects are often hindered by lack of understanding of the underlying dynamics, dependencies and data required to solve a problem.

Below are five steps engineers must take to successfully build AI and ML solutions. 

1. Understand the problem: Before any development, engineers must have a thorough understanding of the problem. The team must identify the dynamics associated with the problem, a list of internal and external dependencies, and data attributes. The methods to achieve this include the ‘5 whys’, mind mapping, online analysis and domain SME discussion.

2. Understand data points: A thorough understanding of existing data is essential as engineers must be able to identify features that could influence the AI/ML model. Common tools and methods to help with this stage include data analysis, statistical methods, data interpolation and extrapolation, feature extraction methods and research.

3. Determine whether AI/ML is appropriate

This is perhaps the most important step in any project development to determine which technologies should be used to solve the problem. Many development teams opt to use AI and ML even when they are not the ideal choice.

Some good questions to ask before you start:

• Can the problem be solved with rule-based-analysis?  
• Is the output strongly influenced by less than 1 to 4 metrics? 
• Are there any existing functions that can facilitate the desired output? 
• Is the problem computationally difficult but has a fixed outcome?
• Is it simple to group the data or series of inputs? 

If the answer to any of the questions above is a strong yes, the project may not be a good fit for AI and ML. 

4. Spend time on feature engineering 

One of the most challenging tasks is establishing effective features that are dependable for ML classification. Based on the nature of a problem, features must be extracted from transformation. This requires significant expertise and application of data engineering concepts.

5. Apply ML algorithms contextually  

Selecting the correct ML algorithm for a problem requires analyzing and using KPIs such as accuracy, precision, recall, MCC (Matthews correlation coefficient), and F scores. There is no ‘minimum data’ formula, as it is highly driven by the complexity of the problem. 

Selecting the right ML algorithm

As with the steps above, it is important to ensure the correct ML algorithm is selected for the success of the entire project. Once the problem, data attributes, and project objectives are thoroughly understood, an algorithm should be chosen. 

Broadly, algorithms fall into four categories:

• Supervised learning
• Semi-supervised learning
• Unsupervised learning
• Reinforcement learning

When datasets are clear and consistent, supervised learning algorithms are a good choice. But, when data is complex or unclear, unsupervised learning algorithms are more appropriate.

It is important to remember that contextual application of ML algorithms is essential to project success.  Your choice of algorithm must be made objectively. Just because you have resources who specialize in an algorithm does not mean it is the best choice. 

Biases in AI based solutions

With emerging technologies, businesses are scrambling to build and adopt AI/ML solutions. One can expect projects to fail owing to any of the below listed reasons:

• Many businesses want to adopt AI/ML simply because they are new and exciting, not because they are the ideal solution to an existing challenge.
• Many businesses do not have the in-house talent (engineers with limited Al/ML skillsets) to enable successful development projects. 
• Inherent bias introduced unknowingly into the data and/or the algorithm is another major issue. 
• The metrics used by engineers to measure AI must be neutral and holistic

Lesson learnt? Have a structured testing process with the right expertise to evaluate the entire process of solution development.  

Kiran Marri is vice president and chief scientist -- digital engineering services at CSS Corp. The views expressed in this column are his own.

---