Promptuna vs. Optuna: Language Models in Hyperparameter Tuning

In recent years, the spotlight has shifted beyond traditional tuning methods like random or grid search. Tools like Optuna have popularized automated hyperparameter optimization, and more recently, large language models (LLMs) such as ChatGPT and GPT-4 have emerged as powerful alternatives for parameter exploration.

Interestingly, fine-tuned models like Code Llama have demonstrated comparable — or even superior — performance to Optuna with significantly fewer trials [arxiv.org]. Other studies suggest that LLMs can rival Bayesian optimization in terms of search efficiency [arxiv.org].

In this post, we’ll explore that idea by using the Titanic dataset to compare three different methods for tuning a LightGBM model:

• Method 1: Training LGBM with default settings
• Method 2: Bayesian optimization using Optuna
• Method 3: Iterative suggestions via natural language prompts to the GPT-4o API

After training with each method, we evaluate accuracy and the confusion matrix, and finally generate a comparison chart of the model performance.

Baseline 1: Training LGBM with Default Settings

First, we train an LGBM model using default settings and evaluate its accuracy and confusion matrix.

• Data Preprocessing: Drop unnecessary columns, fill in missing values, and apply one-hot encoding to categorical variables.
• Model Training: Use lightgbm.LGBMClassifier with default parameters.
• Evaluation: Make predictions on the test set and compute the accuracy and confusion matrix.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix

data = pd.read_csv('train.csv')
data

# Drop unnecessary columns
data = data.drop(['Name','Ticket','Cabin','PassengerId'], axis=1)

# Handle missing values: fill Age with median, Embarked with mode
data['Age'].fillna(data['Age'].median(), inplace=True)
data['Embarked'].fillna(data['Embarked'].mode()[0], inplace=True)

# Convert categorical variables to dummy/one-hot encoded variables
data = pd.get_dummies(data, columns=['Sex','Embarked','Pclass'], drop_first=True)

# Split features and target variable
X = data.drop('Survived', axis=1)
y = data['Survived']

# Split into training and test datasets
X_train, X_test, y_train, y_test = train_test_split(X, y, 
                                                    test_size=0.2, random_state=42)

# Train LightGBM model (with default settings)
from lightgbm import LGBMClassifier

model_def = LGBMClassifier(random_state=42, verbose=-1)
model_def.fit(X_train, y_train)

# Make predictions on the test set
y_pred_def = model_def.predict(X_test)
accuracy_def = accuracy_score(y_test, y_pred_def)
cm_def = confusion_matrix(y_test, y_pred_def)

print(f"Accuracy of LGBM with default settings: {accuracy_def:.4f}")
print("Confusion Matrix (Actual vs Predicted):")
print(pd.DataFrame(cm_def,
                   index=['Actual 0','Actual 1'],
                   columns=['Predicted 0','Predicted 1']))

Accuracy of LGBM with default settings: 0.8324
Confusion Matrix (Actual vs Predicted):
          Predicted 0  Predicted 1
Actual 0           91           14
Actual 1           16           58

Baseline 2: Hyperparameter Optimization with Optuna

Next, we perform hyperparameter optimization using Optuna. In Optuna, we define a search space for each parameter and select the best values through iterative trials. The process is as follows:

• Define the Search Space: Parameters such as learning_rate, max_depth, num_leaves, and n_estimators are sampled using Optuna’s Trial object.
• Objective Function: For each trial, train an LGBM model using the suggested parameters and evaluate its accuracy using either cross-validation or a holdout set.
• Run Optimization: Use study.optimize to run multiple trials (e.g., 30), and identify the parameter set that yields the highest accuracy.
• Evaluate the Best Model: Retrain the LGBM model using the best parameters found, and evaluate its performance and confusion matrix on the test set.

import optuna
from sklearn.model_selection import cross_val_score

# Define the objective function for Optuna
def objective(trial):
    params = {
        'n_estimators': trial.suggest_int('n_estimators', 50, 300),
        'learning_rate': trial.suggest_loguniform('learning_rate', 1e-3, 0.3),
        'num_leaves': trial.suggest_int('num_leaves', 20, 100),
        'max_depth': trial.suggest_int('max_depth', 3, 15),
        'min_data_in_leaf': trial.suggest_int('min_data_in_leaf', 1, 30),
        'subsample': trial.suggest_float('subsample', 0.5, 1.0),
    }
    model = LGBMClassifier(**params, random_state=42)
    # Evaluate accuracy using 3-fold cross-validation
    score = cross_val_score(model, X_train, y_train, cv=3, scoring='accuracy').mean()
    return score

# Run optimization (maximize accuracy)
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=20)

best_params = study.best_params
best_score = study.best_value
print("Best parameters optimized by Optuna:", best_params)
print(f"Best cross-validation accuracy: {best_score:.4f}")

# Retrain the model using the best parameters
model_opt = LGBMClassifier(**best_params, random_state=42, verbose=-1)
model_opt.fit(X_train, y_train)
y_pred_opt = model_opt.predict(X_test)
accuracy_opt = accuracy_score(y_test, y_pred_opt)
cm_opt = confusion_matrix(y_test, y_pred_opt)

print(f"Accuracy after Optuna optimization: {accuracy_opt:.4f}")
print("Confusion Matrix (Actual vs Predicted):")
print(pd.DataFrame(cm_opt,
                   index=['Actual 0','Actual 1'],
                   columns=['Predicted 0','Predicted 1']))

Best parameters optimized by Optuna: {'n_estimators': 256, 'learning_rate': 0.008833356525998749, 'num_leaves': 69, 'max_depth': 3, 'min_data_in_leaf': 23, 'subsample': 0.7021883778414518}
Best cross-validation accuracy: 0.8301
Accuracy after Optuna optimization: 0.8045
Confusion Matrix (Actual vs Predicted):
          Predicted 0  Predicted 1
Actual 0           96            9
Actual 1           26           48

Our Approach: Natural Language Prompt-Based Optimization Using the GPT-4o API

Finally, we test an approach where GPT-4o is used to iteratively suggest hyperparameter settings via natural language prompts. The process works as follows:

• Initial Prompt: We send GPT-4o a prompt containing a summary of the dataset and previous accuracy results, and ask it to propose LGBM hyperparameter settings in JSON format.
• Model Training: We train an LGBM model using the settings suggested by GPT-4o and evaluate its accuracy on the test set.
• Prompt Update: We feed the accuracy result back to GPT-4o with a follow-up prompt like, “The previous accuracy was xx. How can we improve it?”, prompting it to suggest a new configuration.
• Iteration: This cycle is repeated several times until the accuracy stops improving or a trial limit is reached.

import os
os.environ["OPENAI_API_KEY"] = 'your_apikey'

from lightgbm import LGBMClassifier
from sklearn.metrics import accuracy_score
import json

from openai import OpenAI

client = OpenAI()  # Uses the OPENAI_API_KEY environment variable

def propose_params_gpt(prev_score=None):
    import re
    if prev_score is None:
        prompt = (
            "Please suggest hyperparameters in JSON format for an LGBM classifier on the Titanic dataset. "
            "Only include the following keys: n_estimators, learning_rate, num_leaves, max_depth, subsample. "
            "Do not include any other parameters. Return JSON only."
        )
    else:
        prompt = (
            f"The previous accuracy was {prev_score:.4f}. "
            "Please suggest a new set of LGBM hyperparameters for the Titanic dataset in JSON format. "
            "Only include the following keys: n_estimators, learning_rate, num_leaves, max_depth, subsample. "
            "Do not include any other parameters. Return JSON only."
        )

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.7,
        max_tokens=200
    )
    content = response.choices[0].message.content

    # Extract only the JSON portion from the response
    match = re.search(r'\{[\s\S]*\}', content)
    if match:
        json_str = match.group()
        return json.loads(json_str)
    else:
        raise ValueError("Failed to extract JSON from GPT response: " + content)

def evaluate_params(params, X_train, y_train, X_test, y_test):
    """
    Train LGBM using the suggested parameters and return the test accuracy.
    """
    # Filter only valid LGBMClassifier parameters
    allowed_keys = {'n_estimators', 'learning_rate', 'num_leaves', 'max_depth', 'subsample'}
    filtered_params = {k: v for k, v in params.items() if k in allowed_keys}
    model = LGBMClassifier(**filtered_params, random_state=42, verbose=-1)
    model.fit(X_train, y_train)
    preds = model.predict(X_test)
    return accuracy_score(y_test, preds)

def optimize_with_gpt(n_trials, X_train, y_train, X_test, y_test):
    """
    Use GPT-4o to iteratively suggest hyperparameters for n_trials,
    and return the best configuration and its accuracy.
    """
    best_score = 0.0
    best_params = None

    for i in range(1, n_trials + 1):
        # Pass argument by position, not keyword
        params = propose_params_gpt(best_score if i > 1 else None)
        score = evaluate_params(params, X_train, y_train, X_test, y_test)

        print(f"Trial {i}: Accuracy={score:.4f}, Params={params}")

        if score > best_score:
            best_score = score
            best_params = params

    print(f"\nBest Parameters: {best_params}")
    print(f"Best Accuracy: {best_score:.4f}")
    return best_params, best_score

best_params, best_score_gpt = optimize_with_gpt(20, X_train, y_train, X_test, y_test)

print('Best Params:', best_params)
print('Best Accuracy:', best_score_gpt)

Trial 1: Accuracy=0.8436, Params={'n_estimators': 100, 'learning_rate': 0.1, 'num_leaves': 31, 'max_depth': 7, 'subsample': 0.8}
Trial 2: Accuracy=0.8436, Params={'n_estimators': 150, 'learning_rate': 0.05, 'num_leaves': 31, 'max_depth': 10, 'subsample': 0.8}
Trial 3: Accuracy=0.8324, Params={'n_estimators': 1000, 'learning_rate': 0.01, 'num_leaves': 31, 'max_depth': 6, 'subsample': 0.8}
Trial 4: Accuracy=0.8212, Params={'n_estimators': 500, 'learning_rate': 0.01, 'num_leaves': 31, 'max_depth': 7, 'subsample': 0.8}
Trial 5: Accuracy=0.8212, Params={'n_estimators': 150, 'learning_rate': 0.05, 'num_leaves': 31, 'max_depth': 7, 'subsample': 0.8}
Trial 6: Accuracy=0.8212, Params={'n_estimators': 150, 'learning_rate': 0.05, 'num_leaves': 31, 'max_depth': 7, 'subsample': 0.8}
Trial 7: Accuracy=0.8492, Params={'n_estimators': 150, 'learning_rate': 0.05, 'num_leaves': 31, 'max_depth': -1, 'subsample': 0.8}
Trial 8: Accuracy=0.8045, Params={'n_estimators': 150, 'learning_rate': 0.01, 'num_leaves': 31, 'max_depth': 10, 'subsample': 0.8}
Trial 9: Accuracy=0.8045, Params={'n_estimators': 150, 'learning_rate': 0.01, 'num_leaves': 31, 'max_depth': 10, 'subsample': 0.8}
Trial 10: Accuracy=0.8045, Params={'n_estimators': 150, 'learning_rate': 0.01, 'num_leaves': 31, 'max_depth': 7, 'subsample': 0.8}
Trial 11: Accuracy=0.8324, Params={'n_estimators': 200, 'learning_rate': 0.05, 'num_leaves': 31, 'max_depth': 7, 'subsample': 0.8}
Trial 12: Accuracy=0.8045, Params={'n_estimators': 150, 'learning_rate': 0.01, 'num_leaves': 31, 'max_depth': 10, 'subsample': 0.8}
Trial 13: Accuracy=0.8045, Params={'n_estimators': 200, 'learning_rate': 0.01, 'num_leaves': 31, 'max_depth': 7, 'subsample': 0.8}
Trial 14: Accuracy=0.8212, Params={'n_estimators': 150, 'learning_rate': 0.03, 'num_leaves': 31, 'max_depth': 7, 'subsample': 0.75}
Trial 15: Accuracy=0.8212, Params={'n_estimators': 500, 'learning_rate': 0.01, 'num_leaves': 31, 'max_depth': 7, 'subsample': 0.8}
Trial 16: Accuracy=0.8045, Params={'n_estimators': 150, 'learning_rate': 0.01, 'num_leaves': 31, 'max_depth': 10, 'subsample': 0.8}
Trial 17: Accuracy=0.8212, Params={'n_estimators': 500, 'learning_rate': 0.01, 'num_leaves': 31, 'max_depth': 7, 'subsample': 0.8}
Trial 18: Accuracy=0.8212, Params={'n_estimators': 150, 'learning_rate': 0.03, 'num_leaves': 31, 'max_depth': 7, 'subsample': 0.8}
Trial 19: Accuracy=0.8212, Params={'n_estimators': 150, 'learning_rate': 0.05, 'num_leaves': 31, 'max_depth': 7, 'subsample': 0.8}
Trial 20: Accuracy=0.8045, Params={'n_estimators': 150, 'learning_rate': 0.01, 'num_leaves': 31, 'max_depth': -1, 'subsample': 0.8}

Best Parameters: {'n_estimators': 150, 'learning_rate': 0.05, 'num_leaves': 31, 'max_depth': -1, 'subsample': 0.8}
Best Accuracy: 0.8492
Best Params: {'n_estimators': 150, 'learning_rate': 0.05, 'num_leaves': 31, 'max_depth': -1, 'subsample': 0.8}
Best Accuracy: 0.8491620111731844

Accuracy Comparison Chart

Based on the results above, we compare the test accuracy of the three methods. The following code displays the accuracy of each method as a bar chart.

import matplotlib.pyplot as plt

methods = ['Default', 'Optuna Optimized', 'GPT-4o Suggested']
accs = [accuracy_def, accuracy_opt, best_score_gpt]  # Accuracies of Baseline1, Baseline2, Ours

plt.figure(figsize=(6,4))
bars = plt.bar(methods, accs, color=['gray', 'skyblue', 'orange'])

plt.ylim(0, 1)
plt.ylabel('Accuracy')
plt.title('Accuracy Comparison of Each Method')

# Display accuracy value above each bar
for bar in bars:
    yval = bar.get_height()
    plt.text(bar.get_x() + bar.get_width()/2.0, yval + 0.01, f"{yval:.2f}", ha='center')

plt.show()

The parameter configuration proposed by GPT achieved the highest accuracy. At first glance, it might seem like a clear “victory for generative AI,” but it’s too soon to celebrate uncritically. Was it truly the optimal solution — or just a lucky outcome? Now is the time to take a closer look at the structure and behavior behind the results.

Overall Characteristics and Patterns

1. num_leaves Fixed at 31

• All 20 trials used num_leaves = 31, indicating that this parameter was never explored.
• GPT-4o likely defaulted to the standard LightGBM configuration and did not actively vary this value.
• 🔎 Insight: The search space was biased, and the LLM failed to introduce diversity in this dimension.

2. Limited Exploration of learning_rate

• The learning_rate starts at 0.1 (Trial 1), drops to 0.05 (Trial 2), then to 0.01 in most later trials.
• Despite consistently low accuracy (~0.8045) with learning_rate = 0.01, GPT-4o continued to propose it.
• 🔎 Insight: GPT failed to adapt based on feedback, repeating underperforming learning rates without corrective action.

3. Best Accuracy Achieved with max_depth = -1

• Trial 7 reached the highest accuracy (0.8492) with no depth limit.
• Most other depths (7, 10) hovered around 0.82 accuracy.
• 🔎 Insight: For mid-size datasets like Titanic, unrestricted depth may allow better model performance — something GPT managed to capture in only a few trials.

4. Repeated Parameter Sets

• Trials 5, 6, and 19 used identical configurations (Accuracy = 0.8212).
• Trials 8, 9, 12, and 16 also repeated the same parameters (Accuracy = 0.8045).
• 🔎 Insight: GPT-4o showed a lack of exploration strategy, repeatedly suggesting the same underperforming configurations.

5. Consistency in High-Accuracy Configurations, but Limited Flexibility

• n_estimators ranged from 100 to 1000, but optimal results clustered around 150.
• subsample remained fixed at 0.8 in nearly all cases.
• 🔎 Insight: Once a good configuration was found, GPT-4o overly relied on it without actively seeking further improvement.

Quantitative Evaluation

Conclusion

GPT-4o demonstrates value in early-stage tuning by quickly proposing “decent” configurations for simple datasets like Titanic. However:

• The lack of exploration,
• Repetition of known-bad configurations, and
• Absence of strategic adaptation (exploration vs. exploitation)

highlight its limitations as an optimizer.

GPT-4o can serve as a useful tool for quick initial tuning, but serious optimization demands additional layers of memory, diversity, and exploration strategy.

To unlock GPT’s full potential for hyperparameter optimization, it must be paired with an external meta-controller that manages score history, diversity injection, and prompt variation — creating a hybrid system that balances creativity with rigor.