Adversarial AI refers to a class of artificial intelligence systems that are designed to overcome “security measures,” such as authentication protocols, firewalls, and intrusion detection systems. These systems employ machine learning algorithms and techniques to learn from the data and identify vulnerabilities that can be exploited. It is characterized by its ability to use advanced techniques such as generative adversarial networks (GANs), reinforcement learning, and other methods for generating fake input data to deceive AI models and trick them into producing incorrect outputs or misinterpreting inputs. This technology has gained significant attention in recent years due to its potential to cause widespread harm to individuals, organizations, and nations. Adversarial AI can be used for several criminal activities, including hacking, fraud, identity theft, spam, and malware. Therefore, the development of robust and reliable countermeasures against this technology has become a top priority for governments, researchers, and industry leaders alike.

The Contemporary Threat of AI Arms Race

The contemporary threat of an AI arms race is a pressing concern that requires urgent attention. The increasing development of AI technology has led several countries to pursue the creation of powerful autonomous weapon systems that can operate independently without human intervention. The widespread availability of these advanced weapons presents serious risks to global security, especially in the absence of an international agreement to manage them. The increasing number of countries investing in the development of these AI-based arms systems has increased the likelihood of an arms race that could result in a destabilizing effect on the international security and reduce any incentives for countries to negotiate arms control agreements. Furthermore, the development of these advanced weapons raises fundamental ethical and safety issues that must be addressed. Therefore, urgent action needs to be taken to avoid the potential for a catastrophic conflict caused by the AI arms race and promote transparency and cooperation among nations.

In response to the increasing threat of adversarial AI, researchers have been working to develop methods to detect and defend against these attacks. One approach is to use adversarial training, where the AI is trained on examples of both regular and adversarial inputs. This helps the AI to learn to recognize and resist attacks, as it becomes more robust to variations in input. Another approach is to use generative models to create synthetic data that is similar to real-world examples, but contains specific variations that can be used to train a model to recognize adversarial attacks. This is known as data augmentation, as it creates additional variations of the data to improve the generalizability of the model. Additionally, researchers have been exploring the use of explainable AI, which makes it easier to understand how a model makes its predictions, and can help identify when an attack is occurring. These and other techniques are key to maintaining the security of AI systems in the face of escalating adversarial threats.

How it Works

Adversarial AI is designed to operate through a complex system of deep learning algorithms that are trained on rich datasets. These datasets enable adversarial AI models to process and analyze vast amounts of information, recognize patterns, and learn to identify complex structures in the data. The core of adversarial AI lies in its ability to generate false or misleading data that can trick other AI systems into making incorrect predictions or decisions. This process involves the AI system being trained on data that has been intentionally designed to confuse it, making it difficult to identify the real data from the fake. Adversarial AI can also be designed to infiltrate and disrupt the operations of rival AI systems.

By detecting and exploiting the weaknesses of adversaries, adversarial AI systems can initiate attacks through targeted manipulation of data and algorithms. It is crucial to understand the working principles of adversarial AI to develop adequate defense measures. As AI technology advances, the competition between such systems will continue to grow, and the arms race of adversarial AI will only intensify.

Ultimately, the deployment of adversarial AI will have far-reaching ramifications for our society. The arms race between attackers and defenders will fundamentally reshape the nature of cybersecurity and the development of AI. As AI systems become more advanced, they will have the opportunity to learn from their past mistakes and adapt their behavior to circumvent existing defense mechanisms. This creates a cat-and-mouse game where both sides must constantly innovate and improve their technology to stay ahead of the other. However, this race can be exacerbated when development of adversarial AI technology is left unchecked without proper regulation or safeguards. Without adequate oversight, there is a risk that these technologies may be used for malicious purposes, potentially causing serious harm to people or institutions. As such, it is crucial that we consider the potential consequences and implications of this new arms race and take proactive measures to mitigate its negative effects.

The Arms Race in Adversarial AI

The arms race in adversarial AI has given rise to new threats and challenges in the security and defense realms. As AI technology becomes more sophisticated, the potential for adversarial attacks increases.

Sophisticated cyber criminals, nation-states, and terrorists are all seeking ways to exploit AI vulnerabilities to gain a strategic advantage. Governments around the world are investing in AI as part of their national defense strategies, with the goal of developing AI-enabled autonomous weapons systems, cyber warfare capabilities, and intelligence gathering tools. The proliferation of AI is leading to a new era of asymmetrical warfare, where small groups and rogue states can potentially inflict great harm on more powerful nations. Adversarial AI has the potential to disrupt global power relations, increase instability, and bring about new forms of conflict. In this context, international cooperation and regulation are needed to ensure that the development and deployment of AI is done in a responsible and safe manner.

How it Affects the Global Community

Adversarial AI’s arms race is not limited to a single country or region. The global community is already feeling the effects of this phenomenon. The proliferation of AI technologies amplifies the potential for conflict, particularly in the international realm, where nation-states have competing interests. The deployment of adversarial AI by any one of them could quickly escalate tensions and lead to unintended consequences. The arms race has the potential to precipitate global conflict by enabling countries to use AI-driven cyber attacks with unprecedented effectiveness. Moreover, the dangers posed by adversarial AI are not exclusively military. As AI systems become more ubiquitous and more powerful, they will have a profound effect on our daily lives, including transportation, healthcare, finance, and communication. The arms race in adversarial AI has the potential to undermine the international order and disrupt global progress if effective measures are not taken to mitigate its impact.

Different Global Players Involved in the Arms Race

In addition to the United States and China, other nations have also been involved in the arms race for AI technology. Russia, for example, has made significant investments in developing advanced military AI capabilities, and has already deployed autonomous drones in Syria. North Korea has also invested in AI for military applications, despite its limited resources, with a focus on developing AI-powered cyberattack capabilities. Israel is a global leader in developing military AI, and its advanced surveillance and reconnaissance technologies have been put to use in its ongoing conflicts in the Middle East. Similarly, the United Kingdom has developed a variety of AI-powered systems for its military, including a drone swarm designed for remote reconnaissance and attack. The involvement of a growing number of global players in the AI arms race poses significant challenges for maintaining international security and stability. As more nations develop advanced military AI technologies, the risk of accidents, miscalculations, or intentional escalation increases.

Impact to the Adversarial AI Arms Race

Another area that Adversarial AI has been used in is the financial sector for fraud detection. It is well-known that financial institutions are some of the most heavily targeted institutions when it comes to cyber attacks. The use of Adversarial AI in the analysis of financial data has the potential to revolutionize fraud detection. Adversarial AI is capable of identifying patterns and anomalies in financial data that may be invisible to the human eye. The technology enables financial institutions to detect fraudulent activities and accurately predict fraudulent trends before they occur. Furthermore, Adversarial AI algorithms can be integrated with existing fraud management systems to enhance their efficiency, making fraud detection more accurate and cost-effective. The primary benefit of Adversarial AI in financial fraud detection is the ability to significantly reduce false positives and negatives. Adversarial AI can be trained to identify and flag any suspicious financial activities, allowing the financial institution’s fraud management team to investigate and take action.

As the adversarial AI arms race intensifies, its negative implications are becoming increasingly clear. The cost of developing these technologies will certainly be high, diverting resources away from other areas of research and development. Additionally, it is likely that the emergence of highly advanced adversarial AI systems will disrupt global power balances, leading to geopolitical tensions and conflicts. These AI systems could also wreak havoc on economies and financial systems, and pose complex ethical dilemmas around the use of these technologies in warfare.

Furthermore, as these systems become more sophisticated and autonomous, it becomes harder for humans to discern the line between what is ethical and what is not. In the long run, unchecked development of these technologies could pave the way for an AI arms race that could lead to the proliferation of autonomous killing machines, and trigger a catastrophic global conflict. It is, therefore, necessary to ensure that the development and deployment of adversarial AI systems are regulated through a responsible and transparent process.

Consequences for Global Politics and Security

The consequences of the arms race of adversarial AI for global politics and security cannot be underestimated. As the development and deployment of these technologies becomes increasingly widespread, nations will undoubtedly seek to use them to gain strategic advantages over one another. This could lead to a new era of military escalation, as each country tries to outdo the others in terms of technological sophistication.

The use of adversarial AI could lead to destabilizing effects in other areas of international relations, such as trade and diplomacy. For example, countries may be more reluctant to engage in diplomatic negotiations or to trade with one another if they believe that the other party is using adversarial AI to gain an unfair advantage. Ultimately, if left unchecked, the arms race of adversarial AI could have significant and far-reaching consequences for global stability and security, posing a threat to international cooperation and peace.

Personal Privacy and Safety

Another key area of concern is personal privacy and safety. Adversarial AI can be used to create deepfakes and other forms of forged content, which can be used to manipulate public opinion or even cause harm to individuals. For example, deepfakes could be used to create a fake video of a politician making inflammatory remarks, which could then be spread widely on social media.

In addition, adversarial attacks could be used to compromise the security of encrypted communications by manipulating the encryption keys or other aspects of the cryptographic system. This could have serious consequences for individuals and organizations that rely on secure communications for sensitive information.

Overall, the arms race of adversarial AI poses serious challenges to our society, requiring ongoing research and investment in defensive measures to protect against these threats. While AI has the potential to bring many benefits, ensuring that it is developed and used responsibly is essential to safeguarding the public interest.

Economic Impact on AI Development and Regulation

The economic impact of AI regulation is a complex and nuanced issue. While some argue that heavy regulation could stifle innovation and slow development, others suggest that unbridled development could lead to widespread job loss and economic instability. It is important to consider the potential consequences of regulation when looking at the economic impact of AI development. For example, companies who stand to profit from AI development may lobby against strict regulations, while advocates for regulation may prioritize protecting workers and consumers from potential harm. Additionally, the impact of AI on the workforce must be considered.

If AI automation leads to widespread job loss, the economic consequences could be severe. Careful consideration should be given to the balance between innovation and regulation, to ensure that AI is developed in a responsible, sustainable manner that benefits both the economy and society as a whole.

One potential solution to the rapidly escalating arms race of adversarial AI is to focus on creating more resilient AI systems that can withstand attacks from malicious actors. This involves not just strengthening individual systems, but also improving the overall infrastructure surrounding AI development and deployment.

One approach is to incorporate security measures throughout the entire AI life cycle, from data collection to model training to deployment. Another involves developing AI systems that are capable of detecting and defending against adversarial attacks in real time. For instance, AI systems could be trained to recognize unusual or anomalous behavior and take action to mitigate potential threats. Additionally, collaboration between researchers, industry experts, and policymakers will be critical in developing effective solutions to this complex problem. Ultimately, ensuring the safety and security of AI systems will require a multi-faceted approach that addresses technical, social, and ethical considerations.

The Need for Regulation

The implications of adversarial AI are beyond security breaches. As the technology advances, its impact on society may grow exponentially. For example, companies may use adversarial AI to manipulate consumers with targeted advertising leading to unethical marketing practices. Additionally, there are also some long-standing ethical issues associated with AI. AI has the ability to discriminate against certain groups of people, and such potential problems may be amplified by adversarial AI.

Governments are already struggling to regulate AI on many fronts, including privacy and data regulation. Adversarial AI raises additional concerns regarding transparency, accountability, and responsibility. One solution is to create regulatory bodies that include professionals in AI, legal experts, and other relevant stakeholders to set standards and guidelines for the development and deployment of these technologies. It is essential that policymakers take proactive measures to regulate adversarial AI to ensure that this technology is accessible to everyone and operates within ethical and legal boundaries.

The Role of Governments, Institutions, and AI Industry Players

The roles of governments, institutions, and AI industry players are essential in shaping the future of adversarial AI. Governments need to establish regulations and policies that promote ethical AI development to prevent weaponizing AI technology. Institutions can help in advancing research into AI’s robustness and defenses against adversarial attacks. They can also provide training and education to individuals and organizations to better understand how to protect systems from these attacks.

AI industry players can collaborate with governments and institutions to create standardized guidelines for designing and deploying AI systems ethically. They can also incorporate more advanced security and defense mechanisms into their products and services to prevent and mitigate adversarial attacks. A coordinated approach from these players is necessary to ensure the responsible and ethical deployment of AI and to prevent the negative consequences of adversarial AI.

Legal and Ethical Considerations

It is important for developers to ensure that their systems comply with regulations and laws, such as data protection laws, to safeguard users’ data. AI systems must also comply with ethical principles, such as fairness and accountability, to ensure just outcomes. Developers need to consider the impact of adversarial AI on marginalized individuals or groups, such as minority communities, and avoid perpetuating biased outcomes. Furthermore, developers need to consider human values such as respect, dignity, and privacy when developing adversarial AI. Ethical and legal considerations must underpin the development of adversarial AI to prevent the occurrence of various ethical dilemmas and limit potential harm to users.

Potential Ways to Regulate the Arms Race

To regulate the Arms Race, one potential way is for governments to come together and establish international treaties and agreements that outline acceptable behaviors in the development, deployment, and use of artificial intelligence in military applications. This could include regulations on the types of AI that are allowed to be developed, restrictions on certain weapons systems, and requirements for transparency and accountability in the design and operation of AI-powered military technologies. Additionally, implementing measures to ensure that these rules are enforced and adhered to is critical to their effectiveness.

Another potential approach is to increase education and awareness about the risks and benefits of AI in the context of military applications, both among policymakers and the general public. This could help to foster a more informed and nuanced conversation around this emerging technology and its potential impact on global security and stability. Ultimately, successfully regulating the arms race will require a multifaceted approach that engages government, industry, civil society, and other stakeholders to work together towards a common goal of ensuring that AI is used responsibly and ethically in military contexts.

As adversarial AI becomes more advanced and sophisticated, it raises ethical concerns and security risks. The increasing power of adversarial AI models, designed to generate false data or manipulate the input, poses significant security risks as they can easily be used for malicious purposes. These models are capable of generating fake news, deep fakes, and phishing content that can have a detrimental impact on individuals and society as a whole. Furthermore, adversarial AI can be used by bad actors to exploit vulnerabilities in existing AI systems, such as autonomous vehicles and other automated technology. This arms race of adversarial AI presents a challenge for researchers and developers who must stay on top of the latest advances in AI and security in order to keep pace with the attackers. It also raises important questions about the ethical use of AI and the need for regulation. There is a growing need for collaboration and cooperation between stakeholders to mitigate the risks of adversarial AI and ensure that it is used for socially beneficial purposes.

Collaboration between the private and public sector is critical to ensure that our nation’s information security is not compromised. As Adversarial AI gains momentum, we must stay one step ahead, with a firm understanding of how these systems work and the development of techniques to mitigate their potential threats. Only then can we foster security and trust in the digital age.

The adversarial AI arms race is a double-edged sword that poses both threats and opportunities to society. While AI has immense potential to resolve some of the world’s most pressing problems, it can also be weaponized and used to destabilize territories and societies. Therefore, there is a need for proactive measures to prevent the misuse of AI. This includes the establishment of international standards, policies, and regulations that ensure AI is developed and used ethically. Moreover, there is a need for mass awareness and education campaigns to help the public appreciate the risks of AI and to advocate for responsible AI developments. Nonetheless, the adversarial AI arms race is hardly over, and it is likely to escalate in the foreseeable future. The race will be characterized by fast iterations, secrecy, and a lot of unknowns, making it a complex and challenging problem to solve. As such, it is up to industry leaders, policymakers, and civil societies to work collectively and harness the full potential of AI to foster sustainable development without unduly compromising human safety and security.

The post The Arms Race of Adversarial AI appeared first on The Official Blog of Adam DiStefano, M.S., CEH, CISSP, CCSK, CAISS.

The post Leveraging GPT for Authentication: A Deep Dive into a New Realm of Cybersecurity appeared first on The Official Blog of Adam DiStefano, M.S., CEH, CISSP, CCSK, CAISS.

The post Enhancing SIEM with GPT Models: Unleashing the Power of Advanced Language Models in Cyber Security appeared first on The Official Blog of Adam DiStefano, M.S., CEH, CISSP, CCSK, CAISS.

Before we dive in, I think it i important to distinguish between these two approaches in supervised learning. As a reminder, in linear regression, the algorithm learns to identify the linear relationship between input variables and output variables. The goal is to find the best-fitting line that describes the relationship between the input variables and the output variables. This line is determined by minimizing the sum of the squared differences between the predicted values and the actual values. During training, the algorithm is provided with a set of input variables and their corresponding output labels. The algorithm uses this data to learn the relationship between the input and output variables. Once the algorithm has learned this relationship, it can use it to make predictions on new, unseen data.

In classification, the algorithm learns to identify patterns in the input data and assign each input data point to one of several possible categories. The goal is to find a decision boundary that separates the different categories as well as possible. During training, the algorithm is provided with a set of input variables and their corresponding output labels, which represent the categories to which the input data points belong. The algorithm uses this data to learn the relationship between the input variables and the output labels, and to find the decision boundary that best separates the different categories. Once the algorithm has learned this relationship, it can use it to make predictions on new, unseen data. 

Let’s get started.

Background and Context

AllLife Bank is a US bank that has a growing customer base. The majority of these customers are liability customers (depositors) with varying sizes of deposits. The number of customers who are also borrowers (asset customers) is quite small, and the bank is interested in expanding this base rapidly to bring in more loan business and in the process, earn more through the interest on loans. In particular, the management wants to explore ways of converting its liability customers to personal loan customers (while retaining them as depositors).

A campaign that the bank ran last year for liability customers showed a healthy conversion rate of over 9% success. This has encouraged the retail marketing department to devise campaigns with better target marketing to increase the success ratio.

We will attempt to build a model that will help the marketing department to identify the potential customers who have a higher probability of purchasing the loan.

Data Dictionary

• ID: Customer ID
• Age: Customer’s age in completed years
• Experience: #years of professional experience
• Income: Annual income of the customer (in thousand dollars)
• ZIP Code: Home Address ZIP code.
• Family: the Family size of the customer
• CCAvg: Average spending on credit cards per month (in thousand dollars)
• Education: Education Level. 1: Undergrad; 2: Graduate;3: Advanced/Professional
• Mortgage: Value of house mortgage if any. (in thousand dollars)
• Personal_Loan: Did this customer accept the personal loan offered in the last campaign?
• Securities_Account: Does the customer have securities account with the bank?
• CD_Account: Does the customer have a certificate of deposit (CD) account with the bank?
• Online: Do customers use internet banking facilities?
• CreditCard: Does the customer use a credit card issued by any other Bank (excluding All life Bank)?

Methodology

We will start by following the same methodology as we did in our linear regression model: 

1. Data Collection: Begin by collecting a dataset that contains the input features. This dataset will be split into a training set (used to train the model) and a testing set (used to evaluate the model’s performance).
2. Data Preprocessing: Clean and preprocess the data, addressing any missing values or outliers, and scaling the input features to ensure that they are on the same scale.
3. Model Training: Train the logistic regression model on the training dataset. This step involves finding the best-fitting line that minimizes the error between the actual and predicted purchase likelihood. Most programming languages, such as Python, R, or MATLAB, have built-in libraries that simplify this process.
4. Model Evaluation: Evaluate the model’s performance on the testing dataset by comparing its predictions to the actual loan purchases. Common evaluation metrics for classification models include: 
   1. Accuracy: The proportion of correctly classified instances to the total number of instances in the test set.
   2. Precision: The proportion of true positives (correctly classified positive instances) to the total number of predicted positives (instances classified as positive).
   3. Recall: The proportion of true positives to the total number of actual positives in the test set.
   4. F1 score: The harmonic mean of precision and recall, which provides a balance between the two measures.
   5. Area under the receiver operating characteristic curve (AUC-ROC): A measure of the performance of the algorithm at different threshold levels for classification. The AUC-ROC curve plots the true positive rate (recall) against the false positive rate (1-specificity) for different threshold levels.
   6. Confusion matrix: A table that summarizes the actual and predicted classifications for each class. It provides information on the true positives, true negatives, false positives, and false negatives.
5. Model Optimization: If the model’s performance is unsatisfactory, consider feature engineering, adding more data, or using regularization techniques to improve the model’s accuracy.

The dataset used to build this model can be found by visiting my GitHub page.

Data Collection

We will start by importing all our required Python libraries:

#Import NumPy
import numpy as np

#Import Pandas
import pandas as pd
pd.set_option('mode.chained_assignment', None)
pd.set_option("display.max_columns", None)
pd.set_option("display.max_rows", 200)

#Import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline

#Import Seaborn
import seaborn as sns

#Import sklearn libraries
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.model_selection import GridSearchCV
from sklearn.tree import DecisionTreeClassifier
from sklearn import tree
from sklearn.metrics import (
    f1_score,
    accuracy_score,
    recall_score,
    precision_score,
    confusion_matrix,
    roc_auc_score,
    plot_confusion_matrix,
    precision_recall_curve,
    roc_curve,
)

#Beautify Python code
%reload_ext nb_black

#Import warnings
import warnings
warnings.filterwarnings("ignore")

#Import Metrics
from sklearn import metrics

Now we will import the dataset. For this project, I used Google Colab.

#mount and connect Google Drive
from google.colab import drive
drive.mount('/content/drive')

#Import dataset "used_cars_data.csv"
data = pd.read_csv('/content/drive/My Drive/Colab Notebooks/Loan_Modeling.csv')

Data Preprocessing, EDA, and Univariate/Multivariate Analysis

As always, we will start by reviewing the data:

#Return random data sample
data.sample(10)

Next, we will evaluate how may rows and columns are in the dataset:

#Number of rows and columns
print(f'Number of rows: {data.shape[0]} and Number of columns: {data.shape[1]}')

As we can see, there are 5,000 rows and 14 columns.

Next, we will review the datatypes:

#Data type review
data.info()

It does not appear that there is any missing data in the dataset. We can confirm by running:

#Confirming no data is missing
data.isnull().sum()

Let’s see if there is any duplicated data:

#Check for duplicates
data.duplicated().sum()

There is no duplicated data identified. Additionally, the ID column does not offer any added value so we will drop this column.

#Drop ID column
data.drop(['ID'], axis=1, inplace=True)
data.reset_index(inplace=True, drop=True)

Next, we will review the statistical analysis:

#Statistical summary of dataset
data.describe().T

Here is what we found:

Age

• Mean: 45.3
• Minimum Age: 23
• Maximum Age: 67

Experience

• Mean: 20.1
• Minimum Experience: -3
• Maximum Experience: 43

(We will address the negative values below)

Income

• Mean: 73.8
• Minimum Income: 8
• Maximum Income: 224

Family

• Mean: 2.4
• Minimum Family: 1
• Maximum Family: 4

CC Avg

• Mean: 1.9
• Minimum CC Avg: 0
• Maximum CC Avg: 10

Education

• Mean: 1.9
• Minimum Education: 1
• Maximum Age: 3

Mortgage

• Mean: 56.5
• Minimum Mortgage: 0
• Maximum Mortgage: 635

Next. we will review the unique values in the dataset:

#Review unique values
pd.DataFrame(data.nunique())

Zip codes seem to have the most unique values. Since we are dealing with logistic regression which does classifications based on categories, we will want to convert the zip codes into something we can categorize. Since city would most likely return the same number of unique values, we will convert the zip codes to be based on county. This is a mush more macro approach and should reduce the number of unique values in the dataset. This is also a much better approach as all of the zip codes appear to be located in the same state so using the state instead of zip code would not offer much value.

Doing a simple Google search returned a GitHub repo that utilizes a Python library called zipcode that has the ability to map zip codes to specific counties.

#Install the Python zipcode library
!pip install zipcodes

First, we create a list of all the unique values for ZIPCode which will enable us to create an iterative for loop. We will then store these in a dictionary as Zip Code mapped to the county. We will convert the stored values to a string. If the county conversion cannot be identified, we will simply keep the zip code and evaluate the results.

#Import the zipcodes Python package
import zipcodes

#Create a list of the zip codes in the dataset based on these unique values
zip_list = data.ZIPCode.unique()
zipcode_dictionary = {}

for zip in zip_list:
    zip_to_county = zipcodes.matching(zip.astype('str'))
    if len(zip_to_county)==1:

    #Get the county from the zipcodes package
        county = zip_to_county[0].get('county')

    else:
        county = zip
    zipcode_dictionary.update({zip:county})

#Return the dictionary
zipcode_dictionary

The following zip codes were not mapped to the county:

• 92634
• 92717
• 93077
• 96651

We will drop these rows.

#Drop all rows with 92634 zip code
data = data[data["ZIPCode"] != 92634]

#Drop all rows with 92717 zip code
data = data[data["ZIPCode"] != 92717]

#Drop all rows with 93077 zip code
data = data[data["ZIPCode"] != 93077]

#Drop all rows with 96651 zip code
data = data[data["ZIPCode"] != 96651]

Let’s review the shape of the data now:

#Review the shape of the data
data.shape

The data shape has now been reduced by (1) column after dropping the ID column and (44) rows by eliminating zip codes that could not be mapped to a county. We now need to map these counties to the dataset by using the map function. According to the map function “returns a list of the results after applying the given function to each item of a given iterable.”

Next, we will create a new column called County that maps the zip codes in the dataset to the new feature, counties.

#Create new column county that maps the zip codes accordingly
data['County'] = data['ZIPCode'].map(zipcode_dictionary)

We will now convert the newly created county column to a categorical datatype.

#Convert the county column to a category
data['County'].astype('category')

To review the counties by count:

#Value counts by county
data['County'].value_counts()

The top five counties where customers reside are as follows:

• Los Angeles County: 1095
• San Diego County: 568
• Santa Clara County: 563
• Alameda County: 500
• Orange County: 339

It was observed above that there were some negative values in the experience column above that we need to address. We can do a number of things here. We can impute using a measure of central tendency, we could drop the rows, we can replace these with zeros, or we can use the absolute value function. Let’s first understand the impact before we determine which strategy would be best.

#Identify all the rows with negative values for experience
data[data['Experience'] < 0].value_counts().sum()

There are 51 rows with negative values for the experience column. Since it is impossible to have a negative number of years of experience and we do not know if this was a clerical error, we are going to replace those values with zeros. We could also use the absolute value, but we chose to make them 0.

#Replace negative values with zeros
data.loc[data['Experience']<0,'Experience'] = 0

Let’s take a visual look at the continuous data in the dataset:

As we move to univariate analysis, I decided to create a function to make representing this data graphically easier.

#Create a function for univariate analysis (code used from Class Module)
def histogram_boxplot(data, feature, figsize=(12, 7), kde=False, bins=None):
    f2, (ax_box2, ax_hist2) = plt.subplots(
        nrows=2,
        sharex=True,
        gridspec_kw={"height_ratios": (0.25, 0.75)},
        figsize=figsize,
    )  
    sns.boxplot(
        data=data, x=feature, ax=ax_box2, showmeans=True, color="violet"
    )  
    sns.histplot(
        data=data, x=feature, kde=kde, ax=ax_hist2, bins=bins, palette="winter"
    ) if bins else sns.histplot(
        data=data, x=feature, kde=kde, ax=ax_hist2
    )  
    ax_hist2.axvline(
        data[feature].mean(), color="green", linestyle="--"
    ) 
    ax_hist2.axvline(
        data[feature].median(), color="black", linestyle="-"
    )

Additionally, I built a function to help identify outliers that exist in our dataset.

#Create function for outlier identification
def feature_outliers(feature: str, data = data):
    Q1 = data[feature].quantile(0.25)
    Q3 = data[feature].quantile(0.75)
    IQR = Q3 - Q1
    return data[((data[feature] < (Q1 - 1.5 * IQR)) | (data[feature] > (Q3 + 1.5 * IQR)))]

Evaluating the age feature, we see the age feature looks relatively normal and even.

The mean and median ages are approximately 45 years old:

#Mean of age
print(data['Age'].mean())

#Median of age
print(data['Age'].median())

We also identified that there were no outliers in the age feature.

#Evaluate outliers
age_outliers = feature_outliers('Age')
age_outliers.sort_values(by = 'Age', ascending = False)
age_outliers

Looking at the education feature, we see that the mean and median number of years respectively is 1.88 and 2.0 years.

#Mean of education
print(data['Education'].mean())

#Median of education 
print(data['Education'].median())

We will also convert this feature to categorical datatype:

#Convert Education columns to category

data[‘Education’] = data[‘Education’].astype(‘category’, errors = ‘raise’)

Next, we will review the experience feature. The mean experience is 20.1 and the median is 20. This data looks relatively normal. Additionally, there were no outliers.

#Mean of experience
print(data['Experience'].mean())

#Median of experience
print(data['Experience'].median())

#Evaluate outliers
experience_outliers = feature_outliers('Experience')
experience_outliers.sort_values(by = 'Experience', ascending = False)
experience_outliers

The data for the income feature is right skewed.There is approximately $10,000 difference between the mean and median income. Additionally, there are 96 outliers for the income feature. We will not change these as these customers may be in the market for a personal loan.

#Mean of income
print(data['Income'].mean())

#Mean of income
print(data['Income'].median())

#Evaluate outliers
income_outliers = feature_outliers('Income')
income_outliers.sort_values(by = 'Income', ascending = False)
income_outliers.head()
income_outliers.value_counts().sum()

There are 3,435 customers in the dataset that do not report having a mortgage. There are 289 outliers for the mortgage feature. Again, we will leave these as is.

Let’s also evaluate the top 10 zip codes of where our customers reside who do not have a mortgage.

We also observed the mean for the CCAvg feature is 1.9 and the median is 1.5. There were also 320 outliers identified for the CCAvg feature. We will leave this as some customers may apply for personal loans for debt consolidation.

The mean family size is 2.4 and the median is 2.0. We will convert the family column to a categorical datatype.

#Mean of family
print(data['Family'].mean())

#Median of experience
print(data['Family'].median())

#Convert family columns to category
data['Family'].astype('category', errors = 'raise')

The top three counties are:

• Los Angeles County
• San Diego County
• Santa Clara County

We will convert this column to a categorical datatype and drop the Zip Code column.

#Convert County columns to category
data['County'] = data['County'].astype('category', errors = 'raise')

#Drop ZIPCode column
data.drop(['ZIPCode'], axis=1, inplace=True)
data.reset_index(inplace=True, drop=True)

The data showed that only 10.63% of customers in the dataset have a personal loan. Our next step is to convert this feature into a category.

#Percentage of customers with personal loans
percentage = pd.DataFrame(data['Personal_Loan'].value_counts(ascending=False))
took_personal_loan = (percentage.loc[1]/percentage.loc[0] * 100).round(2)
print(f'{took_personal_loan[0]}% of customers have a personal loan.')

#Convert Personal_Loancolumns to category
data['Personal_Loan'] = data['Personal_Loan'].astype('category', errors = 'raise')

We observed that 11.62% of customers have security accounts. We will convert the security accounts to a categorical datatype.

#Percentage of customers with personal loans
percentage = pd.DataFrame(data['Personal_Loan'].value_counts(ascending=False))
took_personal_loan = (percentage.loc[1]/percentage.loc[0] * 100).round(2)
print(f'{took_personal_loan[0]}% of customers have a personal loan.')

There are a few other features we could have conducted our univariate analysis on, however for the sake of brevity, here is the main findings:

• The mean age is 45.3 years old and the median age is 45
• The mean experience is 20.1 and the median age is 20
• The mean income is approximately 
• 64,000 per year. There is approximately $10,000 difference between the mean and median.
• The mean CCAvg is 1.9 and the median is 1.5
• 10.63% of customers have a personal loan
• 67.54% of customers use online banking
• 11.62% of customers have security accounts
• 6.48% of customers have a CD account
• 41.56% of customers have a credit card account
• The top three counties are Los Angeles County, San Diego County, and Santa Clara County
• The mean education is 1.9 and the median is 2.0

We will now create a function to assist in our bivariate analysis:

#Function for Multivariate analysis (code taken from class notes)

def stacked_barplot(data, predictor, target):
    count = data[predictor].nunique()
    sorter = data[target].value_counts().index[-1]
    tab1 = pd.crosstab(data[predictor], data[target], margins=True).sort_values(
        by=sorter, ascending=False
    )
    print(tab1)
    print("-" * 120)
    tab = pd.crosstab(data[predictor], data[target], normalize="index").sort_values(
        by=sorter, ascending=False
    )
    tab.plot(kind="bar", stacked=True, figsize=(count + 5, 6))
    plt.legend(
        loc="lower left", frameon=False,
        plt.legend(loc=“upper left”, bbox_to_anchor=(1, 1))

        plt.show()
    )

Now that we have the function created, let’s look at the breakdown of those customers with personal loans broken down by family size.

We see that the families with 3 kids are the largest demographic with personal loans. Another interesting fact that we identified in our bivariate analysis is that more people in the 60+ age group took the personal loan than those who didn’t. Most people who took the personal loan are between the ages of 30-60.

Below is a breakdown of the continuous values in the dataset in a pair plot:

This helped us identify that the experience column does not appear to offer much value in terms of building the models so we will drop this column. Since age and experience go are so heavily correlated, we do not need this column. We will drop experience and keep age.

#Drop Experience column
data.drop(['Experience'], axis=1, inplace=True)
data.reset_index(inplace=True, drop=True)

Below is a heat map of the numerical representations of the correlation:

Model Building

Now that our data analysis is completed, we will start building some models. We will first start with using a standard logistic regression model as our baseline to see if we can improve upon the results in iterations.

The first step is to make a copy of our original dataset.

#Copy dataset for logistic regression model
data_lr = data.copy()

Now that we are using a clean dataset, we can start building our logistic regression model. To begin, we will drop the dependent variable and use the same one-hot encoding technique we used in our linear regression model. W will encode the county, family, and education features.

Model using sklearn

#Beginning building Logistic Regression Model
x = data_lr.drop(['Personal_Loan'], axis=1)
y = data_lr['Personal_Loan']

#Use OneHot Encoding on county, family, and education
oneHotCols=['County','Education', 'Family']
x = pd.get_dummies(x, columns = oneHotCols, drop_first = True)

Next, we will split our dataset into training and testing data respectively.

# splitting in training and test set
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.3, random_state=1)

We now have 3,476 rows in our training data and 1,490 rows in our testing dataset. Now that it is split, we can effectively fit the model using the libliner solver, predict on the test data, and evaluate the coefficients.

#Build the model
model = LogisticRegression(solver="liblinear", random_state=1)
lg = model.fit(x_train, y_train)

#predicting on test
y_predict = model.predict(x_test)

#Evaluate the coefficients
coef_df = pd.DataFrame(
    np.append(lg.coef_, lg.intercept_),
    index=x_train.columns.tolist() + ["Intercept"],
    columns=["Coefficients"],
)
coef_df.T

What we notice here is that the coefficients of age, securities account, online, credit card, El Dorado County, Fresno County, Humboldt County, Imperial County, Lake County, Los Angeles County, Mendocino County, Merced County, Monterey County, Placer County, Riverside County, Sacramento County, San Benito County, San Bernardino County, San Diego County, San Francisco County, San Joaquin County, San Luis Obispo County, San Mateo County, Santa Barbara County, Santa Cruz County, Shasta County, Siskiyou County, Stanislaus County, Trinity County, Tuolumne County, and Family_2 are negative and an increase in these will lead to decrease in chances they purchase a personal loan.

Let’s evaluate the results on the training dataset:

• True Negatives (TN): Correctly predicted that they do not have personal loan (3,213)
• True Positives (TP): Correctly predicted that they have personal loan (213)
• False Positives (FP): Incorrectly predicted that they have a personal loan (24 falsely predict positive Type I error)
• False Negatives (FN): Incorrectly predicted that they don’t have a personal loan (116 falsely predict negative Type II error)

In evaluating the training performance, we see the accuracy score really well, but the recall is pretty low here.

#Evaluate metrics on the Training Data (Taken from class module)
log_reg_model_train_perf = model_performance_classification_sklearn_with_threshold(lg, x_train, y_train)
print("Training performance:")
log_reg_model_train_perf

The coefficients of the logistic regression model are in terms of log(odd), to find the odds we have to take the exponential of the coefficients. Therefore, odds = exp(b). The percentage change in odds is given as odds = (exp(b) – 1) * 100

#Converting coefficients to odds
odds = np.exp(lg.coef_[0])

#Finding the percentage change
perc_change_odds = (np.exp(lg.coef_[0]) - 1) * 100

#Removing limit from number of columns to display
pd.set_option("display.max_columns", None)

# Adding the odds to a dataframe
pd.DataFrame({"Odds": odds, "Change_odd%": perc_change_odds}, index=x_train.columns).T

• Age: A 1 unit change in Age will decrease the odds of a person buying a personal loan by 0.98 times or a 1.58% decrease in odds of having purchased a personal loan.
• Income: a 1 unit change in the Income will increase the odds of a person having purchased a personal loan by 1.05 times or a 4.99% increase in odds of having purchased a personal loan.
• CCAvg: a 1 unit change in the CCAvg will increase the odds of a person having purchased a personal loan by 1.14 times or a 13.96% increase in odds of having purchased a personal loan.
• Mortgage: a 1 unit change in the mortgage will increase the odds of a person having purchased a personal loan by 1.00 times or a 0.06% increase in odds of having purchased a personal loan.
• Securities_Account: a 1 unit change in the securities_account will decrease the odds of a person having purchased a personal loan by 0.39 times or a 61.46% decrease in odds of having purchased a personal loan.
• CD_Account: a 1 unit change in the CD_account will increase the odds of a person having purchased a personal loan by 26.65 times or a 2565.05% increase in odds of having purchased a personal loan.
• Online: a 1 unit change in the online will decrease the odds of a person having purchased a personal loan by 0.49 times or a 51.36% decrease in odds of having purchased a personal loan.
• Credit Card: a 1 unit change in the Credit Card will decrease the odds of a person having purchased a personal loan by 0.40 times or a 59.35% decrease in odds of having purchased a personal loan.

Other noticable considerations include:

• County_Contra Costa County: a 1 unit change in the County_Contra Costa County will increase the odds of a person having purchased a personal loan by 1.93 times or a 92.56% increase in odds of having purchased a personal loan.
• County_Sonoma County: a 1 unit change in the County_Sonoma County will increase the odds of a person having purchased a personal loan by 1.91 times or a 90.81% increase in odds of having purchased a personal loan.
• County_Sonoma County: a 1 unit change in the County_Sonoma County will increase the odds of a person having purchased a personal loan by 1.91 times or a 90.81% increase in odds of having purchased a personal loan.
• Education_2: a 1 unit change in the Education_2 will increase the odds of a person having purchased a personal loan by 11.91 times or a 1006.28% increase in odds of having purchased a personal loan.
• Education_3: a 1 unit change in the Education_3 will increase the odds of a person having purchased a personal loan by 12.19 times or a 1118.67% increase in odds of having purchased a personal loan.
• Family_3: a 1 unit change in the Family_3 will increase the odds of a person having purchased a personal loan by 4.27 times or a 326.90% increase in odds of having purchased a personal loan.
• Family_4: a 1 unit change in the Family_4 will increase the odds of a person having purchased a personal loan by 3.21 times or a 220.66% increase in odds of having purchased a personal loan.

Plotting the ROC-AUC returns:

#Plot the ROC-AOC
logit_roc_auc_train = roc_auc_score(y_train, lg.predict_proba(x_train)[:, 1])
fpr, tpr, thresholds = roc_curve(y_train, lg.predict_proba(x_train)[:, 1])
plt.figure(figsize=(7, 5))
plt.plot(fpr, tpr, label="Logistic Regression (area = %0.2f)" % logit_roc_auc_train)
plt.plot([0, 1], [0, 1], "r--")
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
plt.title("Receiver operating characteristic")
plt.legend(loc="lower right")
plt.show()

Model Using Optimal Threshold of .12

#Optimal threshold as per AUC-ROC curve
fpr, tpr, thresholds = roc_curve(y_train, lg.predict_proba(x_train)[:, 1])
optimal_idx = np.argmax(tpr - fpr)
optimal_threshold_auc_roc = thresholds[optimal_idx]
print(optimal_threshold_auc_roc)

Plugging this threshold in, we can now see if this improves our metrics:

#Function for confusion matrix with optimal threshold

def confusion_matrix_sklearn_with_threshold(model, predictors, target, threshold=0.1278604841393869):
    pred_prob = model.predict_proba(predictors)[:, 1]
    pred_thres = pred_prob > threshold
    y_pred = np.round(pred_thres)
    cm = confusion_matrix(target, y_pred)
    labels = np.asarray(
        [
            ["{0:0.0f}".format(item) + "\n{0:.2%}".format(item / cm.flatten().sum())]
            for item in cm.flatten()
        ]
    ).reshape(2, 2)
    plt.figure(figsize=(6, 4))
    sns.heatmap(cm, annot=labels, fmt="")
    plt.ylabel("True label")
    plt.xlabel("Predicted label")

• True Negatives (TN): Correctly predicted that they do not have personal loan (2,885)
• True Positives (TP): Correctly predicted that they have personal loan (296)
• False Positives (FP): Incorrectly predicted that they have a personal loan (262 falsely predict positive Type I error)
• False Negatives (FN): Incorrectly predicted that they don’t have a personal loan (33 falsely predict negative Type II error)

Let’s review the score with the newly applied threshold.

#Checking model performance for this model
log_reg_model_train_perf_threshold_auc_roc = model_performance_classification_sklearn_with_threshold(lg, x_train, y_train, threshold=optimal_threshold_auc_roc)
print("Training performance:")
log_reg_model_train_perf_threshold_auc_roc

This significantly improved our recall score but at the expense of our precision.

Model Using Optimal Threshold of .33

#Setting the threshold
optimal_threshold_curve = 0.33

• True Negatives (TN): Correctly predicted that they do not have personal loan (3,078)
• True Positives (TP): Correctly predicted that they have personal loan (248)
• False Positives (FP): Incorrectly predicted that they have a personal loan (69 falsely predict positive Type I error)
• False Negatives (FN): Incorrectly predicted that they don’t have a personal loan (81 falsely predict negative Type II error)

Evaluating the score with the adjusted optimal threshold:

#Metrics with threshold set to 0.33
log_reg_model_train_perf_threshold_curve = model_performance_classification_sklearn_with_threshold(lg, x_train, y_train, threshold=optimal_threshold_curve)
print("Training performance:")
log_reg_model_train_perf_threshold_curve

We successfully increased the precision, but the recall has now dropped. Since we are concerned about recall as that is the best measure for how well our model is predicting positive cases, we see that the model using the .12 threshold performed the best on our training data.

#Training performance comparison
models_train_comp_df = pd.concat(
    [
        log_reg_model_train_perf.T,
        log_reg_model_train_perf_threshold_auc_roc.T,
        log_reg_model_train_perf_threshold_curve.T,
    ],
    axis=1,
)
models_train_comp_df.columns = [
    "Logistic Regression sklearn",
    "Logistic Regression-0.12 Threshold",
    "Logistic Regression-0.33 Threshold",
]
print("Training performance comparison:")
models_train_comp_df

We will now evaluate our model on the testing data.

Model Using sklearn

• True Negatives (TN): Correctly predicted that they do not have personal loan (1,218)
• True Positives (TP): Correctly predicted that they have personal loan (133)
• False Positives (FP): Incorrectly predicted that they have a personal loan (124 falsely predict positive Type I error)
• False Negatives (FN): Incorrectly predicted that they don’t have a personal loan (15 falsely predict negative Type II error)

#Metrics on test data
log_reg_model_test_perf = model_performance_classification_sklearn_with_threshold(lg, x_test, y_test)
print("Test set performance:")
log_reg_model_test_perf

We will see if we can improve the recall scores using the optimal threshold. This has a really decent precision score however.

#Plot test data
logit_roc_auc_test = roc_auc_score(y_test, lg.predict_proba(x_test)[:, 1])
fpr, tpr, thresholds = roc_curve(y_test, lg.predict_proba(x_test)[:, 1])
plt.figure(figsize=(7, 5))
plt.plot(fpr, tpr, label="Logistic Regression (area = %0.2f)" % logit_roc_auc_test)
plt.plot([0, 1], [0, 1], "r--")
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
plt.title("Receiver operating characteristic")
plt.legend(loc="lower right")
plt.show()

Model Using Optimal Threshold of .12

#Creating confusion matrix on test with optimal threshold
confusion_matrix_sklearn_with_threshold(lg, x_test, y_test, threshold=optimal_threshold_auc_roc)

• True Negatives (TN): Correctly predicted that they do not have personal loan (1,218)
• True Positives (TP): Correctly predicted that they have personal loan (133)
• False Positives (FP): Incorrectly predicted that they have a personal loan (124 falsely predict positive Type I error)
• False Negatives (FN): Incorrectly predicted that they don’t have a personal loan (15 falsely predict negative Type II error)

Reviewing the metric scores using the optimal threshold set to 0.12, we see a very good recall score but a lower precision.

#Checking model performance for this model
log_reg_model_test_perf_threshold_auc_roc = model_performance_classification_sklearn_with_threshold(lg, x_test, y_test, threshold=optimal_threshold_auc_roc)
print("Test set performance:")
log_reg_model_test_perf_threshold_auc_roc

Model Using 0.33 Threshold

Lastly, we will evaluate the testing data using a 0.33 threshold to see if we can improve these metrics any further.

#Creating confusion matrix with optimal threshold
confusion_matrix_sklearn_with_threshold(lg, x_test, y_test, threshold=optimal_threshold_curve)

• True Negatives (TN): Correctly predicted that they do not have personal loan (1,311)
• True Positives (TP): Correctly predicted that they have personal loan (105)
• False Positives (FP): Incorrectly predicted that they have a personal loan (31 falsely predict positive Type I error)
• False Negatives (FN): Incorrectly predicted that they don’t have a personal loan (43 falsely predict negative Type II error)

NOTE: Type I errors reduced to 31 from 124, but type II errors increased to 43 from 15.

#Checking model performance for this model
log_reg_model_test_perf_threshold_curve = model_performance_classification_sklearn_with_threshold(
    lg, x_test, y_test, threshold=optimal_threshold_curve
)
print("Test performance:")
log_reg_model_test_perf_threshold_curve

e have successfully improved the precision. However, the recall score has significantly degraded. Additionally the model using the optimal threshold of 0.12 proves to be the strongest model.

#Testing performance 
log_reg_model_test_perf_threshold_curve = model_performance_classification_sklearn_with_threshold(lg, x_test, y_test, threshold=optimal_threshold_curve)
log_reg_model_test_perf_threshold_curve
models_test_comp_df = pd.concat(
    [
        log_reg_model_test_perf.T,
        log_reg_model_test_perf_threshold_auc_roc.T,
        log_reg_model_test_perf_threshold_curve.T,
    ],
    axis=1,
)
models_test_comp_df.columns = [
    "Logistic Regression sklearn",
    "Logistic Regression-0.12 Threshold",
    "Logistic Regression-0.33 Threshold",
]
print("Test set performance comparison:")
models_test_comp_df

We have successfully build a supervised learning classification model using logistic regression to help the marketing department to identify the potential customers who have a higher probability of purchasing a loan. Finding the optimal threshold of 0.12 had the strongest results with a recall of roughly 90% on both the testing and training data and had very strong accuracy scores. In a future post, we will expand this by using decision trees to evaluate how much stronger we can build this classification supervised learning model and provide the business some valuable insights.

The post Using Logistic Regression to Predict Personal Loan Purchase: A Classification Approach appeared first on The Official Blog of Adam DiStefano, M.S., CEH, CISSP, CCSK, CAISS.