From Keywords to Insights: Extracting and Classifying Short-Text Data with LLM

As a data analyst, working with free-text data can be tough. Large Language Models (LLMs) make it easier to extract insights from this data. This post shows how the OpenAI API can help you extract data from search keywords.

Photo by Bruno Kelzer on Unsplash

Not a member yet? Read this article using this free version!

As a data analyst, dealing with free-text data can be challenging. Transforming it into insightful information often requires numerous high-effort steps, and the results may lack accuracy. However, thanks to the advancement of Large Language Models, we can now process free-text data with significantly less effort than before. This post will explore a simple use case of extracting data from search keywords and demonstrate how the OpenAI API can help in accomplishing the task. This post will explore a simple use case of extracting data from search keywords and demonstrate how the OpenAI API can help in accomplishing the task.

Text data is everywhere. At times, it’s structured and normalized, making analysis straightforward. For example, in e-commerce, products are categorized into well-defined groups in the database. When asked about last month’s best-selling category, you can quickly aggregate products by category. These categories are standardized options selected by sellers when listing their products. For analysis, you simply join tables, such as the transaction table with the dimension table containing category information.

However, text data can often be unstructured and difficult to analyze. For instance, when buyers search for products on an e-commerce platform, their entries can range from general product names to specific brands.

To understand this data, we need to extract key information like the product, its category, and the brand. In this post, we’ll discuss how the OpenAI API can assist us in extracting this information from free-text search keywords, enabling us to conduct more detailed analyses.

Intro

Before we begin the project, let’s clearly define the use case and provide a brief introduction to the OpenAI API.

The use case

In this post, we aim to extract valuable information from e-commerce search keywords. Users often input free-form text into search bars, and our goal is to classify these keywords into structured categories like product, product category, and brand. Doing so will allow us to conduct more detailed analyses on search trends and customer preferences.

To illustrate this use case, I have created a toy dataset with 100 rows. Each row contains a unique identifier (id) and the search keyword, followed by the product, product category, and the brand recognized from the keyword. This dataset has been human-labeled, so we can use it to evaluate the model’s performance later.

Dataset of dummy search keywords.

Short intro to OpenAI API

Large Language Models (LLMs), like GPT-3.5 and GPT-4o by OpenAI, are transforming our understanding of natural language. These models generate human-like text, understand complex queries, and have wide applications, from answering questions to writing essays.

OpenAI provides two ways to interact with these models. ChatGPT is a user-friendly chat interface, while the OpenAI API is for developers needing more flexibility, allowing tasks like text generation and information extraction.

In this project, we start with GPT-3.5 for its speed and cost-effectiveness, but may switch to GPT-4o for more challenging tasks, thus balancing cost and performance.

Implementation

The steps to complete the tasks are as follows:

1. Generate and set up the OpenAI API key
2. Design the prompt
3. Import the library and the dataset
4. Make an API request and handle the response
5. Executing the function for a single query
6. Executing the function for a a list of queries
7. Evaluate the model’s accuracy
8. Discuss next steps

Step 1: Generate and Set Up anOpenAI API key

The first step in exploring the models and its usage is to generate an API key from the OpenAI platform. This is accomplished on the API keys page (https://platform.openai.com/api-keys). Before generating the key, log in to the OpenAI platform or sign up if you don’t have an account. Make sure you have a credit balance, which can be verified on the Billing page (https://platform.openai.com/account/billing/overview). After generating the key, remember to copy it and store it somewhere safe.

OpenAI Platform API keys page (23 June, 2024)

There are several methods for using API keys safely. OpenAI provides a comprehensive guide on setting up the API key, which can be found here: OpenAI API Key Setup Guide. This article will use the Set up your API key for a single project method.

Let’s assume we’re working on a project in a directory named openai_project. In this directory, create a .env file to store the API key you generated earlier using the code provided below. If you intend to use version control for the project, make sure you've created a .gitignore file and added .env to it. This step ensures your secrets aren’t accidentally shared via version control.

# .env
OPENAI_API_KEY='your-api-key'

# .gitignore
.env

Step 2: Designing the Prompt

Interacting with AI models like OpenAI’s GPT involves crucial attention to prompt design and structure, as these play a pivotal role in obtaining accurate and useful responses. Two essential components of prompt design are the system prompt and the user prompt.

The system prompt gives the AI a comprehensive understanding of its role and the task at hand. It establishes the context and defines the boundaries within which the AI operates. For instance, in our project, where we aim to extract information from e-commerce search keywords, the system prompt might include instructions to identify the product, its category, and the brand. Since we have a specific list of product categories, we should include all of them in the prompt. For consistency, we will instruct the model to output in JSON format and ensure that the model does not provide any extra text which could compromise the process. Finally, we provide two examples of input and the expected output.

This is what the system prompt will look like:

You are an intelligent assistant tasked with extracting information from search bar inputs on an e-commerce platform. 
Each input will be a search keyword. Extract and format the information as follows:

product: The product the user searched for.
product_category: The category of the product. Use one of these categories or leave it blank if none fit:
- Electronics
- Clothing
- Home Appliances
- Furniture
- Kitchenware
brand: The brand of the product. Leave it blank if no brand is mentioned.

Provide the output in JSON format without any additional commentary or text.

Input example:
Keywords:
Dell Inspiron 15 laptop
leather recliner chair

Output example :
{"keyword": "Dell Inspiron 15 laptop", "product": "Laptop", "product_category": "Electronics", "brand": "Dell"},
{"keyword": "leather recliner chair", "product": "Recliner Chair", "product_category": "Furniture", "brand": ""}

The user prompt comprises the actual data or queries that require processing. It provides specific input to the AI, triggering it to execute the task outlined by the system prompt. In the context of our e-commerce example, the user prompt would contain the search keywords input by the users. This prompt is dynamic and varies with each function call, reflecting the specific search term supplied by the user.

Keywords: {keyword}

A comprehensive introduction to prompt engineering is provided by Pak Sparisoma Viridi in this story:

[embed]Short intro to prompt engineering by examples The art of asking the AI assistant to get the right answermedium.com

Prompt engineering requires iterative improvement in real applications, especially for system prompts in our case. For better readability and maintainability, I suggest storing the prompt in a .txt file outside the main or function file. We can store it in the prompts/search_keywords_system_prompt.txt directory, while defining the user prompt in the code.

With everything set up, we can now start writing the main code. For development purposes, we can create a Jupyter notebook file named search_keyword_extraction.ipynb.

Step 3: Importing Libraries and Dataset

Before starting to build the script using Python (assuming Python is already installed), install the OpenAI library. This library will allow us to send requests to the OpenAI API.

You can install it by using the following command:

pip install --upgrade openai

Next, import the necessary libraries:

from openai import OpenAI
import pandas as pd
import numpy as np
import jsonp

With the libraries in place, we can now import the dataset and examine some samples.

dataset = pd.read_csv('dataset/search_keywords.csv')
print(dataset.head(5))

# id                       keyword       product product_category  brand
#  1                Apple iPad Pro        Tablet      Electronics  Apple
#  2             Canon DSLR camera   DSLR Camera      Electronics  Canon
#  3               wooden wardrobe      Wardrobe        Furniture    NaN
#  4  stainless steel kitchen sink  Kitchen Sink      Kitchenware    NaN
#  5           Asus ZenBook laptop        Laptop      Electronics   Asus

Step 4: Making an API Request and Handling the Response

To make an API request to the OpenAI API and extract valuable data from the response, we will encapsulate the process in the extract_data_from_keyword function below. We will break down this process step by step.

def extract_data_from_keyword(keyword):
    # Load system content and user content
    system_content = load_prompt('prompts/search_keywords_system_prompt.txt')
    user_content = f"Keywords: {keyword}"

    # Initialize the OpenAI API client
    client = OpenAI()

    # Make a request to the OpenAI API to generate a chat completion
    chat_completion = client.chat.completions.create(
        messages=[
            {
                "role": "system",
                "content": system_content
            },
            {
                "role": "user",
                "content": user_content
            }
        ],
        model="gpt-3.5-turbo",
        temperature=0.3,
        max_tokens=4096,
        top_p=0.8,
        response_format={ "type": "json_object" }
    )

    # Extract the completion result and token usage information from the response
    completion = chat_completion.choices[0].message.content
    result = json.loads(completion)
    prompt_tokens_used = chat_completion.usage.prompt_tokens
    completion_tokens_used = chat_completion.usage.completion_tokens

    return result, prompt_tokens_used, completion_tokens_used

• Load System and User Prompts. Initially, load the prompt defined previously. For the system prompt, a specific function has been created to load the text, as shown in the snippet below:

def load_prompt(file_path):
    with open(file_path, 'r') as file:
        return file.read()

• Initialize the OpenAI API Client. The client is used to make requests to the OpenAI service. If the secret key is stored using the default name OPENAI_API_KEY, call the function as OpenAI(). If the name is not default, the key name should be defined in the function.
• Make a Request. Generate a chat completion by making a request. The primary inputs for this process are the model, system, and user prompts. To ensure the output is more consistent, send additional parameters, including setting the temperature, max_token, top_p, and response_format, where the response should be in JSON. To know more about these parameters, I suggest to read below discussion on OpenAI Developer Forum.

[embed]Cheat Sheet: Mastering Temperature and Top_p in ChatGPT API Hello everyone! Ok, I admit had help from OpenAi with this. But what I "helped" put together I think can greatly…community.openai.com

• Extract the Completion Result and Token Usage Information from the Response. Three outputs need to be retrieved. First is the main response, defined as completion in the code, and then the JSON is parsed to get result. The other outputs are the number of tokens used during the prompt and completion process, useful for estimating the cost later. Finally, return the extracted response as the function's result.

Step 5: Executing the Function for a Single Query

Once the function definition is complete, we’ll extract information for a single search keyword query: “Apple iPad Pro”. Below is the code where we call the function and print the result.

keyword = "Apple iPad Pro"
result, prompt_tokens_used, completion_tokens_used = extract_data_from_keyword(keyword)

print(f"Result : {result}")
print(f"Prompt Tokens Used : {prompt_tokens_used}")
print(f"Completion Tokens Used : {completion_tokens_used}")

# Result : {'keyword': 'Apple iPad Pro', 'product': 'iPad Pro', 'product_category': 'Electronics', 'brand': 'Apple'}
# Prompt Tokens Used : 218
# Completion Tokens Used : 29

From the results, it’s clear that the model correctly identified the user was searching for an iPad Pro, even though the dataset labeled it as a tablet, and classified it under Electronics. Furthermore, the user specified the brand name as Apple and recognized by the model. A query like this typically generates 218 tokens for the input and 29 for the result.

Step 6: Executing the Function for a List of Queries

The next step explains how to use this function to get results for a batch of search keyword queries. We’ll use the previously imported dataset, but for testing purposes, we’ll only use the first five rows. The following code calls the function to extract data from each keyword individually and combines them into a dataframe.

df_sample = dataset[['keyword']].head(5)

results = []
total_prompt_tokens_used = 0
total_completion_tokens_used = 0

for keyword in tqdm(df_sample['keyword'], desc="Processing keywords"):
    result, prompt_tokens_used, completion_tokens_used = extract_data_from_keyword(keyword)
    results.append(result)

    total_prompt_tokens_used += prompt_tokens_used
    total_completion_tokens_used += completion_tokens_used

df_result = pd.DataFrame(results)

print(df_result)
print(f"Prompt Tokens Used : {total_prompt_tokens_used}")
print(f"Completion Tokens Used : {total_completion_tokens_used}")

# Result:
#                         keyword       product product_category  brand
# 0                Apple iPad Pro      iPad Pro      Electronics  Apple
# 1             Canon DSLR camera   DSLR Camera      Electronics  Canon
# 2               wooden wardrobe      Wardrobe        Furniture       
# 3  stainless steel kitchen sink  Kitchen Sink      Kitchenware       
# 4           Asus ZenBook laptop        Laptop      Electronics   Asus
#
# Prompt Tokens Used : 1092
# Completion Tokens Used : 151

The first five rows of search queries also yield satisfactory results. For these samples, we have categories such as Furniture and Kitchenware. It’s important to note that for items 2 and 3, no brand is recognized from the keyword because the user did not mention any brand in the search term.

Step 7: Model Evaluation

The final step of this post is model evaluation. We’ll generate results for all queries in the dataset and compare them with the actual data. The evaluation will focus on the accuracy of each keyword’s classification into product categories. Besides overall accuracy, we’ll also compute the precision, recall, and F1 score for each category.

Here are the performance evaluation results, with the complete code available in the repository:

Accuracy: 0.96

          Category Precision Recall F1 Score
0      Electronics      1.00   1.00     1.00
1        Furniture      1.00   1.00     1.00
2      Kitchenware      1.00   0.83     0.91
3         Clothing      1.00   1.00     1.00
4  Home Appliances      0.75   1.00     0.86

The overall accuracy is commendable at 0.96. Most categories have perfect precision, recall, and F1 scores, with exceptions being Kitchenware and Home Appliances.

In practice, perfect scores are rare. Often, it’s necessary to decide which metric to prioritize: precision or recall. I suggest readers familiarize themselves with these concepts and consider the risks of favoring one over the other. For instance, is it acceptable to conclude an analysis with relatively low precision but high recall?

Reflecting on these factors is crucial for knowing how to improve the model, whether through another prompt engineering or the selection of a different model for increased accuracy.

Additional Step: Cost Estimation

You may wonder why the function returns the number of tokens used in both the prompt and the completion. This information helps us estimate the cost of running this exercise, particularly if we plan to deploy it on a scheduler or use it in a production stage.

You can monitor the total usage and cost of your OpenAI account on the Usage page. However, this information is cumulative and aggregated by date and model used. To estimate the cost for each run, we need to calculate it ourselves.

The pricing for each model varies based on input and output tokens. As of the date this post was created (23 June 2024), the pricing for GPT-3.5 Turbo is $0.0005 per 1K input tokens and $0.0015 per 1K output tokens.

By combining this information with the output from the API response, we can estimate the cost of running a batch of search keywords.

Conclusion

This story has outlined a method for extracting information from a search keyword on an e-commerce platform. The goal is to replace traditional methodologies that require complex scripting or large training data to obtain similar information and classifications. The actual value emerges when insights are generated from this data:

• What is the most searched product over a specific period?
• Which search provides a higher conversion rate — with brand or without brand?
• Based on user searches, should we expand the platform’s focus categories?

While this post demonstrates a relatively simple case, such an approach can potentially impact stakeholder’s decision through insightful analysis. In practice, more complex solutions may be needed to yield better results. Thus, numerous intriguing approaches, such as prompt engineering techniques and more complex high-level methods, are available for exploration. Let’s save those for another time.

Complete codes can be found in this repository.

[embed]GitHub - miqbalrp/openai-projects: A collection of project that exploring the usage of OpenAI API… A collection of project that exploring the usage of OpenAI API for cases in data & analysis. - miqbalrp/openai-projectsgithub.com

Let’s connect with me in **LinkedIn**.