Machine Learning + Symbolic Reasoning: A Quarkus story on Artificial Intelligence by Mario Fusco + Nicole Prentzas +

Why not Machine Learning alone?

What is Machine Learning? ML is the thing suggesting to buy laundry detergent together with your new washing machine because they're statistically related ...

ML is the thing suggesting to buy laundry detergent together with your new washing machine because they're statistically related ... … or more often the one thinking that, since you want to buy a washing machine, you decided to start up a whole new laundry. What is Machine Learning?

No content

It’s all about (fancy) naming

Why not Machine Learning + Symbolic Reasoning ? ML to find statistical correlations and discover new patterns SR to filter away the ones not making sense under a business point of view

Why not Machine Learning + Symbolic Reasoning ? ML to find statistical correlations and discover new patterns SR to filter away the ones not making sense under a business point of view

Why not Machine Learning + Symbolic Reasoning ? ML to find statistical correlations and discover new patterns SR to filter away the ones not making sense under a business point of view Your business domain knowledge lives here!

No content

No content

No content

No content

No content

Would you allow an hallucinated blue Caterpillar to decide about your business?

No content

General Machine Learning algorithms problems ❖ Accountability ➢ Who is responsible for the decision taken by a ML algorithm? The creator of the ML-model? The service provider? The final user?

General Machine Learning algorithms problems ❖ Accountability ➢ Who is responsible for the decision taken by a ML algorithm? The creator of the ML-model? The service provider? The final user? ❖ Transparency ➢ While traditional algorithms can be examined line by line, arriving at an exact understanding of their internal dynamics and the reasons for a certain result, ML algorithms automatically extracts knowledge from the data, forming an idea of the problem based on statistical correlations known only to it. ➢ Trying to understand the reason that drove to a decision simply looking at the ML-model would be like trying to know why someone thinks in a certain way by dissecting their brain.

General Machine Learning algorithms problems ❖ Accountability ➢ Who is responsible for the decision taken by a ML algorithm? The creator of the ML-model? The service provider? The final user? ❖ New definition of malfunctioning ➢ Given their statistical nature, ML algorithms can hallucinate or make other spectacular mistakes even when they work very well. ➢ These mistakes cannot be fixed in the same way of a software bug. ❖ Transparency ➢ While traditional algorithms can be examined line by line, arriving at an exact understanding of their internal dynamics and the reasons for a certain result, ML algorithms automatically extracts knowledge from the data, forming an idea of the problem based on statistical correlations known only to it. ➢ Trying to understand the reason that drove to a decision simply looking at the ML-model would be like trying to know why someone thinks in a certain way by dissecting their brain.

Why not fixing LLM’s hallucinations using another LLM to validate the outcome of the first?

Why not fixing LLM’s hallucinations using another LLM to validate the outcome of the first? … this reminds me of a 20 years old joke …

Mixing LLM and rule engine for a safer airline chatbot LLM - (Mistral 7B) Implement a chatbot to collect information about the customer and his flight Rule engine - (Drools) Calculate customer’s refund based on precise business rules session data

Mixing LLM and rule engine for a safer airline chatbot LLM - (Mistral 7B) Implement a chatbot to collect information about the customer and his flight Rule engine - (Drools) Calculate customer’s refund based on precise business rules session data Extract customer data from chat Extract flight data from chat Data gathering complete data incomplete data incomplete State machine

DEMO TIME! https://github.com/mariofusco/quarkus-drools-llm

Mixing LLM and rule engine for a safer airline chatbot

No content

Machine Learning is a new paradigm Rule Engine Rules Data Answers A rule based program produces Answers matching Data against Rules

Machine Learning is a new paradigm Rule Engine Rules Data Answers A rule based program produces Answers matching Data against Rules Machine Learning Answers Data Rules A machine learning system discovers Rules comparing Data and expected Answers

Why not mixing them? Rule Engine Rules Data Answers Machine Learning Answers Data Rules

Leveraging the strengths of both entities ML (sub-symbolic AI) + High adaptability and flexibility + Handling complexity, vast amounts of data + Automatic rule extraction - Lack of interpretability - Data dependence Rule-engine (symbolic AI) + Speed and reliability + Effective with well-structured problems + Interpretable - Scalability - Lack of adaptability - Knowledge acquisition Machine Learning Rule-engine Hybrid AI + →

Symbolic Reasoning Knowledge base Inference engine Question Answer Symbolic AI (i.e. Drools) Human input

Hybrid AI = Symbolic Reasoning + Machine Learning Knowledge base Inference engine Question Answer Symbolic AI (i.e. Drools) Knowledge base Inference engine Question Answer Data ML learning process Hybrid AI (Drools + ML) Human input Human input

Hybrid AI = Symbolic Reasoning + Machine Learning Knowledge base Inference engine Question Answer Symbolic AI (i.e. Drools) Knowledge base Inference engine Question Answer Data ML learning process Hybrid AI (Drools + ML) Human input Human input Hybrid AI: more versatile AI system that can efficiently handle complex scenarios, adjust to changing conditions, and improve performance and accuracy.

ML learning process overview ● Business rules (KB) from data. ● Step 1 - “Initialize-rules”: ○ Add decision-rules to cover all instances in the input dataset. ● Step 2 - “Improve-rules”: ○ Identify rules that erroneously support some data. ○ Add decision-rules to fix the errors. ● We used a revised version of the OneR algorithm to learn decision-rules from data.

The OneR algorithm ● OneR, short for "One Rule", is a simple, yet accurate, classification algorithm that generates one rule for each predictor/feature in a dataset, and then selects the rule with the smallest total error as its "one rule" (Berry M and Linoff, 2000)(*). ● Revised version of OneR: the rule with the highest accuracy (smallest error) and highest coverage is selected. We have introduced a “score” metric as the product of the accuracy and coverage of a rule, to ease this selection. (*) F. Alam and S. Pachauri, “Comparative Study of J48, Naive Bayes and One-R Classification Technique for Credit Card Fraud Detection using WEKA,” Adv. Comput. Sci. Technol. , vol. 10, no. 6, 2017.

OneR example ● One frequency table for each predictor against the target variable ● For each condition (i.e. outlook=sunny) the most frequent (majority) class is assigned to the rule: ● The rules with the highest score is selected as the one rule for the predictor

ML learning process ● Step 1 - “Initialize-rules”: ○ Decision-rules to cover all instances in the input dataset ● Step 2 - “Improve-rules”: ○ Find rules that erroneously support some data. ○ Find a new rule for this subset of data. ○ Fix existing rule and create a new rule for this subset of data. step 1 step 2

Drools KB from data Example ● Dataset: diabetes early stage ● Feature selection: ○ polyuria ○ polydipsia ○ gender where age ~ middle age ● Prediction: positive / negative Source: https://www.kaggle.com/code/therealsampat/early-stage-diabetes-prediction

Step 1: initial set of rules rule "rule 0" when $a : DiabetesDatapoint( polydipsia == 'Yes' ) then $a.setPrediction( 'positive' ); update( $a ); end rule "rule 1" when $a : DiabetesDatapoint( polydipsia == 'No' ) then $a.setPrediction( 'negative' ); update( $a ); end KB Assessment: >> Coverage = 100% >> Errors = 28/186 (15%)

Step 2: improve initial ruleset rule "rule 0" when $a : DiabetesDatapoint( polydipsia == 'Yes' ) then $a.setPrediction( 'positive' ); update( $a ); end rule "rule 1" when $a : DiabetesDatapoint( polydipsia == 'No' ) then $a.setPrediction( 'negative' ); update( $a ); end KB Assessment: >> Coverage = 100% >> Errors = 17/186 (7%) rule "rule 2" when $a : DiabetesDatapoint( gender == ‘Female’, polydipsia == 'No' ) then $a.setPrediction( 'positive' ); update( $a ); end ... rules “rule N”

Conclusions ❖ ML is only a fraction of the whole AI story ➢ The general goal of artificial intelligence is mimicking the human intellect and behaviors ➢ There are different ways to achieve this goal, letting the machine learning it from data is only one of them ❖ Machine Learning doesn’t replace Symbolic Reasoning: they are complementary technologies ➢ ML and SR are different technologies with their own pros and cons ➢ Often both the flexibility and human friendliness of ML and the strictness and transparency of SR are required in the same intelligent application and you can leverage and integrate the strengths of both

Our projects ● Quarkus + LLM + Drools - https://github.com/mariofusco/quarkus-drools-llm ● Rules creation from data - https://github.com/nprentza/droolsML ● Neuro-symbolic AI - https://en.wikipedia.org/wiki/Neuro-symbolic_AI ● Approaches in Using Generative AI for Business Automation: The Path to Comprehensive Decision Automation - https://medium.com/@pierrefeillet/approaches-in-using-generative-ai-for-business-automat ion-the-path-to-comprehensive-decision-3dd91c57e38f ● Explainable Machine Learning via Argumentation - https://www.researchgate.net/publication/372688199_Explainable_Machine_Learning_via_ Argumentation References