The post Risks of Chatbot Adoption: Protecting AI Language Models from Data Leakage, Poisoning, and Attacks appeared first on The Official Blog of Adam DiStefano, M.S., CEH, CISSP, CCSK, CAISS.
]]>Natural language models are pre-trained on vast amounts of data from various sources, including websites, articles, and user-generated content. Sensitive information, when inadvertently embedded, often leads to data leakage or privacy concerns when the model generates text based on this information.
Data leakage occurs when unauthorized exposure or access of sensitive or confidential data happens during the process of training or deploying machine learning models. This can happen due to various reasons such as a lack of proper security measures, errors in coding, or intentional malicious activity. Additionally, data leakage can compromise the privacy and security of the data, leading to potential legal and financial implications for businesses. It can also lead to biased or inaccurate AI models, as the leaked data may contain information that is not representative of the larger population.
In late March of 2023, ChatGPT alerted users of an identified flaw that enabled other users to view portions of conversations users had with the chatbot. OpenAi confirmed that a vulnerability in their redis-py open-source library was the cause data leak and subsequently, “During a nine-hour window on March 20, 2023, another ChatGPT user may have inadvertently seen your billing information when clicking on their own ‘Manage Subscription’ page,” according to an article posted on HelpNetSecurity. The article went on to say that OpenAi uses “Redis to cache user information in their server, Redis Cluster to distribute this load over multiple Redis instances, and the redis-py library to interface with Redis from their Python server, which runs with Asyncio.”
Earlier this month, three incidents of data leakage occurred at Samsung as a result of using ChatGPT. Dark Reading reported that “the first incident as involving an engineer who passed buggy source code from a semiconductor database into ChatGPT, with a prompt to the chatbot to fix the errors. In the second instance, an employee wanting to optimize code for identifying defects in certain Samsung equipment pasted that code into ChatGPT. The third leak resulted when an employee asked ChatGPT to generate the minutes of an internal meeting at Samsung.” Samsung has responded by limiting ChatGPT usage internally and placing controls on employees from asking questions of ChatGPT that were larger than 1,024 bytes.
Data poisoning refers to the intentional corruption of an AI model’s training data, leading to a compromised model with skewed predictions or behaviors. Attackers can inject malicious data into the training dataset, causing the model to learn incorrect patterns or biases. This vulnerability can result in flawed decision-making, security breaches, or a loss of trust in the AI system.
I recently read a study entitled “TrojanPuzzle: Covertly Poisoning Code-Suggestion Models” that discussed the potential for an adversary to inject training data crafted to maliciously affect the induced system’s output. With tools like OpenAi’s Codex models and GitHub CoPilot, this could be a huge risk for organizations leveraging code suggestion models. Using basic methods for attempting poisoning data is detectable by static analysis tools that can remove such malicious inputs from the training set, the study shows that there are more sophisticated ways that allow malicious actors to go undetected.
Using the technique coined TROJANPUZZLE works by injecting malicious code into the training data in a way that is difficult to detect. The malicious code is hidden in a puzzle, which the code-suggestion model must solve in order to generate the malicious payload. The attack works by first creating a puzzle that is composed of two parts: a harmless part and a malicious part. The harmless part is used to lure the code-suggestion model into solving the puzzle. The malicious part is hidden in the puzzle and is only revealed after the harmless part has been solved. Once the code-suggestion model has solved the puzzle, it is then able to generate the malicious payload. The malicious payload can be anything that the attacker wants, such as a backdoor, a denial-of-service attack, or a data exfiltration attack.
Model inversion attacks attempt to reconstruct input data from model predictions, potentially revealing sensitive information about individual data points. The attack works by feeding the model a set of input data and then observing the model’s output. With this information, the attacker can infer the values of the input data that were used to generate the output.
For example, if a model is trained to classify images of cats and dogs, an attacker could use a model inversion attack to infer the values of the pixels in an image that were used to classify the image as a cat or a dog. This information is then be used to identify the objects in the image or to reconstruct the original image.
Model inversion attacks are a serious threat to the privacy of users of machine learning models. They can infer sensitive information about users, such as their medical history, financial information, or location. As a result, it is important to take steps to protect machine learning models from model inversion attacks.
Here is a great walk-thru of exactly how a model inversion attack works. The post demonstrates the approach given in a notebook found in the PySyft repository.
Membership inference attacks determine whether a specific data point was part of the training set, which can expose private user information or leak intellectual property. The attack queries the model with a set of data samples, including both those that were used to train the model and those that were not. The attacker then observes the model’s output for each sample and uses this information to infer whether the sample was used to train the model.
For example, if a model is trained to classify images of cats and dogs, an attacker would a membership inference attack to infer whether a particular image was used to train the model. The attacker would do this by querying the model with a set of images, including both cats and dogs, and observing the model’s output for each image. If the model classifies the images as a cat or dog if it was used to train the model, then the attacker is able to infer that the image was used to train the model.
Membership inference attacks are a serious threat to the privacy of users of machine learning models. They are leveraged to infer sensitive information about users, such as their medical history, financial information, or location.
The adoption of chatbots and other AI language models such as ChatGPT can greatly enhance business processes and customer experiences. However, it also comes with new risks and challenges. One major risk is the potential for data leakage and privacy concerns. As discussed, these can compromise the security and accuracy of AI models. Another risk is data poisoning, where malicious actors can intentionally corrupt an AI model’s training data. This ultimately leads to flawed decision-making and security breaches. Finally, model inversion and membership inference attacks can reveal sensitive information about users.
To mitigate these risks, businesses should implement access controls. They should also use the most modern and secure data encryption techniques. Lastly, seek to leverage data handling procedures, regular monitoring and testing, and incorporate human oversight into the machine learning process. Using differential privacy and a secure deployment environment can help protect machine learning models from these threats. It is crucial that businesses stay vigilant and proactive as they continue to adopt and integrate AI technologies into their operations.
The post Risks of Chatbot Adoption: Protecting AI Language Models from Data Leakage, Poisoning, and Attacks appeared first on The Official Blog of Adam DiStefano, M.S., CEH, CISSP, CCSK, CAISS.
]]>The post NLP Query to SQL Query with GPT: Data Extraction for Businesses appeared first on The Official Blog of Adam DiStefano, M.S., CEH, CISSP, CCSK, CAISS.
]]>Natural Language Processing, or NLP, is a branch of artificial intelligence that focuses on enabling machines to understand and interact with human language. In simpler terms, NLP is the ability of machines to read, understand, and generate human language. NLP allows machines to process and analyze vast amounts of natural language data, such as text, speech, and even gestures, and converts them into structured data that is used for analysis and decision-making, through a combination of algorithms, machine learning, and linguistics. For example, a machine using NLP might analyze a text message and identify the sentiment behind it, such as whether the message is positive, negative, or neutral. Or it might identify key topics or entities mentioned in the message, such as people, places, or products.
NLP uses a combination of algorithms, statistical models, and machine learning to analyze and understand human language. Below are the basic steps involved in the NLP process:
SELECT COUNT(*) FROM orders WHERE order_date >= '2022-03-01' AND order_date < '2022-04-01';
This SQL query would retrieve the number of rows (orders) where the order date falls within the month of March, and return the count of those rows. Executives who desire to have these results traditionally rely on skilled database administrators to craft the desired query. These DBA’s then need to validate that the data meets the needs and requirements that were requested. This is a time consuming process as the requests can be much more complex than the example above.
Using ChatGPT to extract insights from databases can provide numerous benefits to businesses. Here are some of the key advantages:
Before we get started, it is important to note that this is simply a proof of concept application. We will be building a simple application to convert a natural language query into an SQL query to extract sales data from an SQL database. Since it is simply a proof of concept, we will be using a SQL database in memory. In production, you would want to connect directly to the enterprise database.
This project can be found on my GitHub.
The first step for developing this application is to ensure you have an API key from OpenAPI.
To get a developer API key from OpenAI, you need to sign up for an API account on the OpenAI website. Here’s a step-by-step guide to help you with that process:
IMPORTANT: Make sure you keep your API key secure, as it is a sensitive piece of information that can be used to access your account and make requests on your behalf. Don’t share it publicly or include it in your code directly. Store it in a separate file or use environment variables to keep it secure.
This project was created using Jupyter notebook. You can install Jupyter locally as a standalone program on your device. To learn how to install Jupyter, visit their website here. Jupyter also comes installed on Anaconda and you can use the notebook there. To learn more about Anaconda, visit their documentation here. Lastly, you can use Google Colab to develop. Google Colab, short for Google Colaboratory, is a free, cloud-based Jupyter Notebook environment provided by Google. It allows users to write, execute, and share code in Python and other supported languages, all within a web browser. You can start using Google Colab by visiting here.
Note: You must have a Google account to use this service.
For this project, the following Python libraries were used:
#Import Libraries import openai import os import pandas as pd import sqlalchemy #Import these libraries to setup a temp DB in RAM and PUSH Pandas DF to DB from sqlalchemy import create_engine from sqlalchemy import text
For this project, I have created a text file to pass my API key to avoid having to hard code my key into my code. We could have set it up as an environment variable, but we would need to associate the key each time we begin a new session. This is not ideal. It is important to note that the text file must be in the same directory as the notebook to use this method.
#Pass api.txt file with open('api.txt', 'r') as f: openai.api_key = f.read().strip()
Next, we will use the pandas library to evaluate the data. We start by creating a dataframe from the dataset and reviewing the first five rows.
#Read in data df = pd.read_csv("sales_data_sample.csv") #Review data df.head()
This code snippet creates a SQLAlchemy engine that connects to an in-memory SQLite database. Here’s a breakdown of each part:
create_engine
: This is a function from SQLAlchemy that creates an engine object, which establishes a connection to a specific database.'sqlite:///memory:'
: This is a connection string that specifies the database type (SQLite) and its location (in-memory). The triple forward slash (///
) is used to denote an in-memory SQLite database.echo=True
: This is an optional argument that, when set to True
, enables logging of generated SQL statements to the console. It can be helpful for debugging purposes.
#Create temp DB temp_db = create_engine('sqlite:///memory:', echo = True)
In this step, we will use the to_sql
method from the pandas library to push the contents of a DataFrame (df
) to a new SQL table in the connected database.
#Push the DF to be in SQL DB data = df.to_sql(name = "sales_table", con = temp_db)
This code snippet connects to the database using the SQLAlchemy engine (temp_db
) and executes a SQL query to get the sum of the SALES
column from the Sales
table. We will also review the output. Here’s a breakdown of the code:
with temp_db.connect() as conn:
: This creates a context manager that connects to the database using the temp_db
engine. It assigns the connection to the variable conn
. The connection will be automatically closed when the with
block ends.results = conn.execute(text("SELECT SUM(SALES) FROM Sales"))
: This line executes a SQL query using the conn.execute()
method. The text()
function is used to wrap the raw SQL query string, which is "SELECT SUM(SALES) FROM Sales"
. The query calculates the sum of the SALES
column from the Sales
table. The result of the query is stored in the results
variable.
#Connect to SQL DB with temp_db.connect() as conn: results = conn.execute(text("SELECT SUM(SALES) FROM Sales")) #Return Results results.all()
This code snippet defines a Python function called create_table_definition
that takes a pandas DataFrame (df
) as input and returns a string containing a formatted comment about an SQLite SQL table named Sales
with its columns.
# SQLite SQL tables with their properties: # ----------------------------------------- # Employee (ID, Name, Department_ID) # Department (ID, Name, Address) # Salary_Payments (ID, Employee_ID, Amount, Date) # ----------------------------------------- #Create a function for table definitions def create_table_definition(df): prompt = """### sqlite SQL table, with its properties: # # Sales({}) # """.format(",".join(str(col) for col in df.columns)) return prompt
To review the output:
#Review results print(create_table_definition(df))
#Prompt Function def prompt_input(): nlp_text = input("Enter desired information: ") return nlp_text #Validate function prompt_input()
This function defines a Python function called combined
that takes a pandas DataFrame (df
) and a string (query_prompt
) as input and returns a combined string containing a formatted comment about the SQLite SQL table and a query prompt.
#Combine these functions into a single function def combined(df, query_prompt): definition = create_table_definition(df) query_init_string = f"###A query to answer: {query_prompt}\nSELECT" return definition + query_init_string
Here, we grab the NLP input and insert the table definitions.:
#Grabbing natural language nlp_text = prompt_input() #Inserting table definition (DF + query that does... + NLP) prompt = combined(df, nlp_text)
openai.Completion.create()
method from the OpenAI API to generate a response using the GPT-3 language model. The specific model used here is ‘text-davinci-002’. The prompt for the model is generated using the combined(df, nlp_text)
function, which combines a comment describing the SQLite SQL table (based on the DataFrame df
) and a comment describing the SQL query to be written. Here’s a breakdown of the method parameters:model='text-davinci-002'
: Specifies the GPT-3 model to be used for generating the response, in this case, ‘text-davinci-002’.prompt=combined(df, nlp_text)
: The prompt for the model is generated by calling the combined()
function with the DataFrame df
and the string nlp_text
as inputs.temperature=0
: Controls the randomness of the model’s output. A value of 0 makes the output deterministic, selecting the most likely token at each step.max_tokens=150
: Limits the maximum number of tokens (words or word pieces) in the generated response to 150.top_p=1.0
: Controls the nucleus sampling, which keeps the probability mass for the top tokens whose cumulative probability exceeds the specified value (1.0 in this case). A value of 1.0 includes all tokens in the sampling, so it is effectively equivalent to using greedy decoding.frequency_penalty=0
: Controls the penalty applied based on token frequency. A value of 0 means no penalty is applied.presence_penalty=0
: Controls the penalty applied based on token presence in the input. A value of 0 means no penalty is applied.stop=["#", ";"]
: Specifies a list of tokens that, if encountered by the model, will cause the generation to stop. In this case, the generation will stop when it encounters a “#” or “;”.The openai.Completion.create()
method returns a response object, which is stored in the response
variable. The generated text can be extracted from this object using response.choices[0].text
.
#Generate GPT Response response = openai.Completion.create( model = 'text-davinci-002', prompt = combined (df, nlp_text), temperature = 0, max_tokens = 150, top_p = 1.0, frequency_penalty = 0, presence_penalty = 0, stop = ["#", ";"] )
Finally, we right a function to format the response from the GPT application:
#Format response def handle_response(response): query = response['choices'][0]['text'] if query.startswith(" "): query = 'SELECT' + query return query
Running the following snippet will return the desired NLP query to SQL query input:
#Get response handle_response(response)
Your output should now look something like this:
"SELECT * FROM Sales WHERE STATUS = 'Shipped' AND YEAR_ID = 2003 AND QTR_ID = 3\n
The post NLP Query to SQL Query with GPT: Data Extraction for Businesses appeared first on The Official Blog of Adam DiStefano, M.S., CEH, CISSP, CCSK, CAISS.
]]>