From Baseline to Top 10%: A Practical Kaggle Competition Playbook

Most people join a Kaggle competition, grab a tutorial XGBoost notebook, tune a few hyperparameters, and wonder why they’re stuck in the bottom half. The mistake isn’t the model choice — it’s treating a Kaggle competition like a single-model problem.

Top competitors think in pipelines. Every decision, how you encode categories, which features you engineer, how you validate, how you blend compounds.

A 0.3% gain here and a 0.4% gain there add up to the difference between the 60th percentile and the top 5%. This blog is to help you go from baseline to the top 10% on Kaggle.

“The model is the last 20% of the work. The first 80% is understanding your data well enough to give the model something worth learning from.”

Ai generated image

Four principles that separate medal winners from the rest:

Understand the metric first. Macro F1? AUC? RMSE? Every metric has implications for your CV strategy and how you threshold predictions. Read the evaluation section of the competition before you write a single line of code.

EDA before any modelling. Distributions, nulls, class imbalance, leaky features — catch these early or pay for it in wasted submissions later.

CV is your ground truth. Your local cross-validation score must correlate with the public leaderboard. If it doesn’t, your validation setup is broken and every “improvement” you think you’re making could be making things worse.

Diversity beats raw power. Three models that make different kinds of errors will outperform a single perfect model whenever blended correctly.

Step 1 — Build a Solid Baseline Fast

Your baseline has one job: give you a reliable starting score to beat. Don’t over-engineer it. Get something running in under 30 minutes, submit, and use that number as your anchor for every experiment that follows.

For tabular competitions, a single XGBoost with default parameters and basic label encoding is all you need to start.

import numpy as np
import pandas as pd
from sklearn.model_selection import StratifiedKFold
from sklearn.metrics import f1_score
from sklearn.preprocessing import LabelEncoder
from xgboost import XGBClassifier

train = pd.read_csv("/kaggle/input/.../train.csv")
test  = pd.read_csv("/kaggle/input/.../test.csv")

# Encode target
le_target = LabelEncoder()
y = le_target.fit_transform(train["target"])

# Label encode all categorical columns naively
cat_cols = train.select_dtypes("object").columns.tolist()
for col in cat_cols:
    le = LabelEncoder()
    train[col] = le.fit_transform(train[col].astype(str))
    test[col]  = le.transform(test[col].astype(str))

X      = train.drop(columns=["id", "target"])
X_test = test.drop(columns=["id"])

# 5-fold stratified CV
skf   = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
oof   = np.zeros((len(X), len(le_target.classes_)))
preds = np.zeros((len(X_test), len(le_target.classes_)))

for fold, (tr_idx, va_idx) in enumerate(skf.split(X, y), 1):
    model = XGBClassifier(n_estimators=500, tree_method="hist",
                          random_state=42)
    model.fit(X.iloc[tr_idx], y[tr_idx], verbose=False)
    oof[va_idx] = model.predict_proba(X.iloc[va_idx])
    preds      += model.predict_proba(X_test) / 5
    score = f1_score(y[va_idx], oof[va_idx].argmax(1), average="macro")
    print(f"Fold {fold}: {score:.5f}")

print(f"OOF Macro F1: {f1_score(y, oof.argmax(1), average='macro'):.5f}")

Submit your baseline immediately. Your public LB score tells you whether your CV is calibrated correctly. If CV shows 0.87 but LB shows 0.72, your validation fold is leaking data or the train/test distributions differ significantly — fix this before doing anything else.

Also, double-check the obvious things people always miss: make sure you’re loading train.csv and test.csv as separate files (not the same file twice), and that test doesn’t contain the target column. These mistakes waste hours.

Step 2 — EDA That Actually Informs Features

Most people do EDA for the sake of it — printing .describe(), making a heatmap, moving on. Useful EDA answers specific questions that directly change how you model.

What does the target distribution look like? Class imbalance requires stratified sampling at a minimum, and possibly class weights or oversampling. For regression, heavy right-tailed data usually require a log transformation of the target itself.

Which features separate the classes best? Plot each numeric feature’s distribution overlaid by target class. Features with strong visual separation are your highest-value raw inputs — build more features from them first.

Are there near-duplicate or highly correlated features? Correlation above 0.95 between two features adds noise without information. Drop one or merge them.

Does train and test come from the same distribution? This one is critical and almost nobody checks it. Run an adversarial validation:

from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

train_adv = train.drop(columns=["target", "id"]).copy()
test_adv  = test.drop(columns=["id"]).copy()
train_adv["is_test"] = 0
test_adv["is_test"]  = 1
combined = pd.concat([train_adv, test_adv], ignore_index=True)
for col in combined.select_dtypes("object").columns:
    combined[col] = LabelEncoder().fit_transform(combined[col].astype(str))
X_adv = combined.drop(columns=["is_test"])
y_adv = combined["is_test"]
auc = cross_val_score(
    RandomForestClassifier(n_estimators=100, random_state=42),
    X_adv, y_adv, cv=5, scoring="roc_auc"
).mean()
print(f"Adversarial AUC: {auc:.4f}")
print("No shift" if auc < 0.6 else " Distribution shift detected!")

If the adversarial AUC is above 0.6, some features look very different between train and test. Those features will hurt your model’s generalization — identify them and consider dropping them.

Step 3 — Feature Engineering Is Your Real Edge

In tabular competitions, feature engineering contributes more to your final score than any hyperparameter tuning ever will. A well-crafted feature that encodes domain knowledge is worth ten times more than switching from XGBoost to LightGBM.

The workflow is simple: engineer a feature, check its importance after one training run, keep it if it appears in the top half of importances, drop it if it doesn’t. Never add features blindly and hope for the best.

We learned this the hard way in the irrigation competition. Our first improved model added 13 new features — VPD proxy, ET proxy, evapotranspiration estimates, and several ratio features. The score went down. The feature importance chart told the story clearly: Crop_Growth_Stage, Mulching_Used, Soil_Moisture, and Wind_Speed_kmh dominated everything. Features like ET_Proxy and VPD_Proxy had near-zero importance and were adding pure noise.

The fix was to cut the noise and engineer more interactions, specifically around the four features that actually mattered:

def add_features(df):
    df = df.copy()

    # High-signal stress indices
    df["Heat_Stress"]      = df["Temperature_C"] * (100 - df["Humidity"])
    df["Water_Balance"]    = df["Rainfall_mm"] - df["Previous_Irrigation_mm"]
    df["Temp_Sq"]          = df["Temperature_C"] ** 2
    df["High_Stress"]      = (
        (df["Temperature_C"] > df["Temperature_C"].median()) &
        (df["Humidity"] < df["Humidity"].median())
    ).astype(int)
    df["log_Rainfall_mm"]  = np.log1p(df["Rainfall_mm"])

    # Crop_Growth_Stage interactions — the most important feature by far
    df["Stage_Season"]     = (df["Crop_Growth_Stage"].astype(str) + "_" +
                               df["Season"].astype(str))
    df["Irrigation_Stage"] = (df["Irrigation_Type"].astype(str) + "_" +
                               df["Crop_Growth_Stage"].astype(str))
    df["Stage_Temp"]       = (df["Crop_Growth_Stage"].astype(str) + "_" +
                               pd.cut(df["Temperature_C"], bins=4).astype(str))
    df["Stage_Moisture"]   = (df["Crop_Growth_Stage"].astype(str) + "_" +
                               pd.cut(df["Soil_Moisture"], bins=4).astype(str))
    df["Stage_Wind"]       = (df["Crop_Growth_Stage"].astype(str) + "_" +
                               pd.cut(df["Wind_Speed_kmh"], bins=3).astype(str))

    # Mulching_Used — 2nd most important, cross it with everything
    mulch = pd.to_numeric(df["Mulching_Used"], errors="coerce").fillna(0)
    df["Mulch_HeatStress"] = mulch * df["Heat_Stress"]
    df["Mulch_Temp"]       = mulch * df["Temperature_C"]
    df["Mulch_Stage"]      = (mulch.astype(str) + "_" +
                               df["Crop_Growth_Stage"].astype(str))

    # Wind — was missing entirely from the baseline
    df["Wind_Temp"]        = df["Wind_Speed_kmh"] * df["Temperature_C"]
    df["Wind_Evap_Proxy"]  = df["Wind_Speed_kmh"] * (100 - df["Humidity"]) / 100
    df["log_Wind"]         = np.log1p(df["Wind_Speed_kmh"])

    # Categorical interactions
    df["Crop_Season"]      = (df["Crop_Type"].astype(str) + "_" +
                               df["Season"].astype(str))
    df["Source_Region"]    = (df["Water_Source"].astype(str) + "_" +
                               df["Region"].astype(str))

    return df

train_fe = add_features(train)
test_fe  = add_features(test)

A quick reference for the types of features that consistently work in tabular competitions:

1. Ratio features — like rain divided by area — are useful when two raw features interact multiplicatively, where neither alone tells the full story.
2. Polynomial features like temp² capture non-linear effects of continuous variables that tree models can technically learn but benefit from having pre-computed.
3. Interaction strings — concatenating crop type with season into a single categorical — are especially powerful for CatBoost’s native categorical handling, creating fine-grained segments the model wouldn’t discover on its own.
4. Bin + cross features take a continuous variable, discretize it into buckets with pd.cut, then cross it with a categorical — a great way to capture threshold effects.
5. Log transforms with np.log1p are almost always worth applying to right-skewed distributions like rainfall, revenue, or counts.
6. Domain flag features — binary columns derived from domain rules like "temperature above median AND humidity below median" — cleanly segment groups that your model would otherwise have to rediscover from scratch.

Step 4 — Proper Cross-Validation Is Your North Star

A good CV setup that correlates with the leaderboard is worth more than any fancy model. If your CV doesn’t track the leaderboard, you are flying blind — every “improvement” might be making things worse in ways you can’t see until the final reveal.

Use StratifiedKFold for classification, KFold for regression, and GroupKFold when rows are related (time series, user-level data). Always use at least 5 folds.

The most important and most commonly violated rule: fit all preprocessing inside the fold on the training split only. Target encoders, scalers, imputers — all of these must be fit on the training fold and only applied (not refit) on the validation fold and test set.

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
import category_encoders as ce

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
oof_xgb  = np.zeros((len(X), n_classes))
test_xgb = np.zeros((len(X_test), n_classes))
print(f"{'Fold':>4} | {'XGB F1':>8} | {'Best Iter':>9}")
print("-" * 28)
for fold, (tr_idx, va_idx) in enumerate(skf.split(X, y), 1):
    X_tr, X_va = X.iloc[tr_idx].copy(), X.iloc[va_idx].copy()
    y_tr, y_va = y[tr_idx], y[va_idx]
    # Fit preprocessing on training fold only - prevents leakage
    pipe     = Pipeline([
        ("te",      ce.TargetEncoder(cols=cat_cols, smoothing=15)),
        ("imputer", SimpleImputer(strategy="median")),
    ])
    X_tr_enc = pipe.fit_transform(X_tr, y_tr)   # fit on train fold
    X_va_enc = pipe.transform(X_va)              # apply to val
    X_te_enc = pipe.transform(X_test)            # apply to test
    model = XGBClassifier(**xgb_params)
    model.fit(X_tr_enc, y_tr,
              eval_set=[(X_va_enc, y_va)],
              verbose=False)
    oof_xgb[va_idx] = model.predict_proba(X_va_enc)
    test_xgb       += model.predict_proba(X_te_enc) / skf.n_splits
    score = f1_score(y_va, oof_xgb[va_idx].argmax(1), average="macro")
    print(f"{fold:>4} | {score:>8.5f} | {model.best_iteration:>9}")
oof_f1 = f1_score(y, oof_xgb.argmax(1), average="macro")
print(f"\nOOF Macro F1: {oof_f1:.5f}")

Step 5 — Multi-Model Ensembling

The single biggest score jump in most Kaggle competitions comes from ensembling diverse models. The keyword is diverse. Three models that make different kinds of errors will reliably outperform three models that are slight variations of the same algorithm.

XGBoost, CatBoost, and LightGBM are all gradient boosted trees, but they differ enough in their split-finding strategy, regularization approach, and categorical handling to complement each other meaningfully. Run all three inside the same fold loop so they see identical train/validation splits:

oof_xgb, oof_cb, oof_lgb    = [np.zeros((len(X), n_classes)) for _ in range(3)]
test_xgb, test_cb, test_lgb = [np.zeros((len(X_test), n_classes)) for _ in range(3)]

for fold, (tr_idx, va_idx) in enumerate(skf.split(X, y), 1):
    X_tr, X_va = X.iloc[tr_idx].copy(), X.iloc[va_idx].copy()
    y_tr, y_va = y[tr_idx], y[va_idx]
    # XGBoost - target encoding pipeline
    pipe     = Pipeline([("te", ce.TargetEncoder(cols=cat_cols)), ("imp", SimpleImputer())])
    X_tr_xgb = pipe.fit_transform(X_tr, y_tr)
    X_va_xgb = pipe.transform(X_va)
    X_te_xgb = pipe.transform(X_test)
    xgb = XGBClassifier(**xgb_params)
    xgb.fit(X_tr_xgb, y_tr, eval_set=[(X_va_xgb, y_va)], verbose=False)
    oof_xgb[va_idx] = xgb.predict_proba(X_va_xgb)
    test_xgb       += xgb.predict_proba(X_te_xgb) / skf.n_splits
    # CatBoost - native categorical handling, no encoding needed
    X_tr_cb = prep_catboost(X_tr)
    X_va_cb = prep_catboost(X_va)
    X_te_cb = prep_catboost(X_test)
    cat_idx = [X_tr_cb.columns.get_loc(c) for c in cat_cols]
    cb = CatBoostClassifier(**cat_params)
    cb.fit(X_tr_cb, y_tr, cat_features=cat_idx,
           eval_set=(X_va_cb, y_va), verbose=False)
    oof_cb[va_idx] = cb.predict_proba(X_va_cb)
    test_cb       += cb.predict_proba(X_te_cb) / skf.n_splits
    # LightGBM - leaf-wise growth, fast, strong on large datasets
    X_tr_lgb = prep_lgbm(X_tr)
    X_va_lgb = prep_lgbm(X_va)
    X_te_lgb = prep_lgbm(X_test)
    lgbm = lgb.LGBMClassifier(**lgb_params)
    lgbm.fit(X_tr_lgb, y_tr, eval_set=[(X_va_lgb, y_va)],
             callbacks=[lgb.early_stopping(75, verbose=False)])
    oof_lgb[va_idx] = lgbm.predict_proba(X_va_lgb)
    test_lgb       += lgbm.predict_proba(X_te_lgb) / skf.n_splits
    s_xgb = f1_score(y_va, oof_xgb[va_idx].argmax(1), average="macro")
    s_cb  = f1_score(y_va, oof_cb[va_idx].argmax(1),  average="macro")
    s_lgb = f1_score(y_va, oof_lgb[va_idx].argmax(1), average="macro")
    print(f"Fold {fold} | XGB: {s_xgb:.5f} | CB: {s_cb:.5f} | LGB: {s_lgb:.5f}")

Once you have OOF predictions from all three models, find the optimal blend weights. A simple grid search works, but Optuna’s TPE sampler finds better weights in far fewer evaluations — especially important for a 3-way blend where the search space is a 2D simplex:

import optuna
optuna.logging.set_verbosity(optuna.logging.WARNING)

def objective(trial):
    w_xgb = trial.suggest_float("w_xgb", 0.0, 1.0)
    w_cb  = trial.suggest_float("w_cb",  0.0, 1.0 - w_xgb)
    w_lgb = 1.0 - w_xgb - w_cb
    blend = w_xgb * oof_xgb + w_cb * oof_cb + w_lgb * oof_lgb
    return f1_score(y, blend.argmax(1), average="macro")
study = optuna.create_study(direction="maximize",
                             sampler=optuna.samplers.TPESampler(seed=42))
study.optimize(objective, n_trials=300, show_progress_bar=False)
best  = study.best_params
w_xgb = best["w_xgb"]
w_cb  = best["w_cb"]
w_lgb = 1.0 - w_xgb - w_cb
print(f"Best OOF F1 : {study.best_value:.5f}")
print(f"XGB: {w_xgb:.3f} | CB: {w_cb:.3f} | LGB: {w_lgb:.3f}")

Step 6 — Stacking Multiple Solution Notebooks

Blending averages model outputs with fixed weights. Stacking goes further — it trains a meta-model on the OOF predictions of your base models, letting it learn which model to trust for which type of sample. This is the technique behind many top-3 finishes.

The most powerful form is notebook-level stacking. You take the best public notebooks from the competition discussion tab, run them to get their OOF predictions, and stack those together with your own. You’re getting months of other people’s feature engineering work for free.

from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

# Stack OOF predictions from all base models as features
stack_train = np.hstack([oof_xgb, oof_cb, oof_lgb])    # (n_train, n_classes * 3)
stack_test  = np.hstack([test_xgb, test_cb, test_lgb])  # (n_test,  n_classes * 3)
# Option A: Logistic Regression meta-model - simple, rarely overfits
scaler         = StandardScaler()
stack_train_sc = scaler.fit_transform(stack_train)
stack_test_sc  = scaler.transform(stack_test)
meta_lr  = LogisticRegression(C=0.1, max_iter=1000, random_state=42)
meta_lr.fit(stack_train_sc, y)
lr_preds = meta_lr.predict_proba(stack_test_sc)
# Option B: LightGBM meta-model - more expressive, use with care
meta_lgb  = lgb.LGBMClassifier(n_estimators=500, num_leaves=15,
                                 learning_rate=0.05, random_state=42)
meta_oof  = np.zeros((len(stack_train), n_classes))
meta_test = np.zeros((len(stack_test),  n_classes))
for fold, (tr, va) in enumerate(skf.split(stack_train, y)):
    meta_lgb.fit(stack_train[tr], y[tr],
                  eval_set=[(stack_train[va], y[va])],
                  callbacks=[lgb.early_stopping(50, verbose=False)])
    meta_oof[va] = meta_lgb.predict_proba(stack_train[va])
    meta_test   += meta_lgb.predict_proba(stack_test) / skf.n_splits
meta_f1 = f1_score(y, meta_oof.argmax(1), average="macro")
print(f"Meta-model OOF F1: {meta_f1:.5f}")
# Final: blend stacking output with the weighted ensemble
w_stack  = 0.3
final    = (w_stack * meta_test +
            (1 - w_stack) * (w_xgb * test_xgb + w_cb * test_cb + w_lgb * test_lgb))
final_preds = le.inverse_transform(final.argmax(1))

For notebook-level stacking: download 3–4 high-scoring public notebooks, save their OOF predictions and test predictions to CSVs, load them in, and add them as extra columns in your stack_train and stack_test matrices. Then retrain the meta-model on the whole combined stack.

Step 7 — Optuna Hyperparameter Tuning

Manual hyperparameter tuning is slow and biased toward your intuitions. Optuna’s TPE sampler finds better configurations in 100 trials than most people find in days of manual search. The critical rule: only tune after your feature engineering is stable. Tuning noisy features is wasted compute.

def xgb_objective(trial):
    params = dict(
        n_estimators        = trial.suggest_int("n_estimators", 3000, 10000),
        learning_rate       = trial.suggest_float("learning_rate", 0.005, 0.05, log=True),
        max_depth           = trial.suggest_int("max_depth", 4, 8),
        min_child_weight    = trial.suggest_int("min_child_weight", 1, 10),
        subsample           = trial.suggest_float("subsample", 0.6, 1.0),
        colsample_bytree    = trial.suggest_float("colsample_bytree", 0.5, 1.0),
        reg_alpha           = trial.suggest_float("reg_alpha", 1e-3, 5.0, log=True),
        reg_lambda          = trial.suggest_float("reg_lambda", 1e-3, 5.0, log=True),
        objective           = "multi:softprob",
        num_class           = n_classes,
        tree_method         = "hist",
        device              = "cuda",
        random_state        = 42,
        early_stopping_rounds = 50,
    )

    # Use 3-fold CV inside Optuna for speed
    cv  = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
    oof = np.zeros((len(X), n_classes))

    for tr_idx, va_idx in cv.split(X, y):
        pipe     = Pipeline([("te", ce.TargetEncoder(cols=cat_cols)), ("imp", SimpleImputer())])
        X_tr_enc = pipe.fit_transform(X.iloc[tr_idx], y[tr_idx])
        X_va_enc = pipe.transform(X.iloc[va_idx])
        m = XGBClassifier(**params)
        m.fit(X_tr_enc, y[tr_idx], eval_set=[(X_va_enc, y[va_idx])], verbose=False)
        oof[va_idx] = m.predict_proba(X_va_enc)

    return f1_score(y, oof.argmax(1), average="macro")

study = optuna.create_study(direction="maximize",
                             sampler=optuna.samplers.TPESampler(seed=42))
study.optimize(xgb_objective, n_trials=100)

print("Best params:", study.best_params)
print(f"Best F1: {study.best_value:.5f}")

Step 8 — Read the Leaderboard Like a Map

The public leaderboard is only scored on roughly 30% of the test set. Final standings are determined by the private leaderboard score on the remaining 70%. Many teams shake down dramatically at the end because they over-optimised for the public LB — chasing noise in 30% of the data while the 70% that actually matters told a different story.

Always select two final submissions. One that maximises your public LB score, and one that you trust based on your local CV. The CV-trusted submission often wins on private. Kaggle gives you two selections for a reason.

Trust your CV over the leaderboard. If CV and LB disagree, the right response is to fix your CV setup — not to submit more and chase LB noise.

Read the discussion tab every day. Top competitors share insights, external data discoveries, and dataset quirks throughout the competition. This community knowledge is often worth more than another hyperparameter tuning run.

The original dataset trick. Playground Series competitions are synthetically generated from a real original dataset. Find it in the data sources section or by searching Kaggle datasets, add it to your training data with pd.concat([train, original_data]), and you frequently get a free 0.5–1% gain just from more training samples.

The Kaggle Competition Checklist

Before your final submission, run through this list. Each item is either a potential free gain or a potential disaster averted.

1. Does your CV correlate with the leaderboard? This is the foundation. If it doesn’t, every decision you make is based on wrong feedback. Fix this before anything else.

2. Is your pipeline free of data leakage? All preprocessing — target encoding, scaling, imputation — must be fit inside the fold on the training split only, never on the full dataset before splitting.

3. Have you checked feature importance? Run it after your first full training pass. Remove features with near-zero importance, and build more interactions around the top signals.

4. Are you running at least three diverse models? XGBoost, CatBoost, and LightGBM is the standard minimum for tabular competitions. Each handles categories and regularization differently enough to complement the others.

5. Are your blend weights optimised? Don’t use equal weights or a naive 21-point grid. Use Optuna’s TPE sampler — it finds better weights in fewer trials, especially for 3-way blends.

6. Have you added the original dataset? Playground Series competitions are generated from a real source dataset. Find it, concat it to your training data, and enjoy a near-free score bump.

7. Have you selected two final submissions? One that maximises your public LB score, and one you trust based on your local CV. They should not be the same submission.

8. Have you run adversarial validation? Train a classifier to predict whether a row is from the train or the test. If it does well, your features look different across the two sets — and those features will hurt your generalization.

9. Have you tried stacking? Training a meta-model on your OOF predictions consistently gives a small but reliable lift on top of a simple blend.

10. Are you reading the discussion tab daily? Community members share dataset quirks, external data sources, and winning insights throughout the competition. This is often worth more than another tuning run.

Kaggle is a game of compounding marginal gains. No single trick wins competitions. What wins is a disciplined, systematic pipeline where every component — your data, your features, your CV, your models, your blend — is working in concert.

Good luck — and may your CV always correlate with the private LB :)

This blog is published in partnership with UselessAI.in. Follow to read more about data and AI technologies and product experiences.