From OOM Errors to Working Model: Fine-Tuning Gemma 4 E2B Step-by-Step using Unsloth

A practical, no-BS guide to fine-tuning Google’s latest multimodal model using Unsloth on limited hardware.

Gemma 4 fine-tuned using Unsloth on Kaggle — image created by author using Imagen 4 text-to-image

Introduction

Just days after the release of Gemma 4, I wanted to test how far I could push it on a free Kaggle GPU. The promise was exciting: a modern multimodal model capable of reasoning, coding, and even object detection — all available with open weights.

Reality, however, was less friendly.

I tried first with transformers library, really to modify my former Notebooks where I fine-tuned models like Gemma 2 or Llama 3 for a sentiment analysis task using transformers, trl, and SFTTrainer. Out-of-memory errors, broken quantization paths, and incompatible PEFT layers made it clear that fine-tuning Gemma 4 is not as straightforward as earlier models (at least for me).

This article walks through what worked, what failed, and how I finally managed to fine-tune Gemma 4 E2B using Unsloth.

The problems

When experimenting first with transformers framework, inference went well. When switching to fine-tuning, I encountered OOM problems due to few factors. Fine-tuning requires significantly more memory than inference because gradients and optimizer states must be stored. Multimodal towers increases the required memory. Even batch size 1 would require vast amount of memory. Also, I had problems with PEFT incompatibilities, not yet fully adapted. Gemma 4 uses custom layers. Gemma4ClippableLinear is not supported. LoRA injection failed. On Kaggle, bitsandbytes and transformers are mismatching.

Why Unsloth worked?

Unsloth optimizes memory and training. It supports Gemma models and prepared for the release, with documented support for fine-tuning as well. It is faster (up to 60% faster) and uses less VRAM.

[embed]Fine-tuning Google Gemma with Unsloth: Transforming Natural Language into Code - AI Blog What's happening: Fine-tuning Google Gemma with Unsloth: Transforming Natural Language into Codeiartificial.blog

[embed]Gemma 4 Fine-tuning Guide | Unsloth Documentation Train Gemma 4 by Google with Unsloth.unsloth.ai

The working setup

Initialize the model

I start by installing the latest unsloth library on my Kaggle T4 x 2 environment:

!pip install -U -q unsloth

The libraries needed for working with unsloth and performing fine-tuning:

from unsloth import FastLanguageModel
import torch
from datasets import load_dataset, Dataset
from trl import SFTTrainer, SFTConfig

Upon running the above cell:

🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
🦥 Unsloth Zoo will now patch everything to make training faster!

Next, I initialized the model using FastLanguageModel from unsloth:

max_seq_length = 512  # start small; scale up after it works

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "google/gemma-4-E2b-it",
    max_seq_length = max_seq_length,
    load_in_4bit = False,     # MoE QLoRA not recommended, dense 27B is fine
    load_in_16bit = True,     # bf16/16-bit LoRA
    full_finetuning = False,
)

Upon running the above cell:

==((====))==  Unsloth 2026.4.2: Fast Gemma4 patching. Transformers: 5.5.0.
   \\   /|    Tesla T4. Num GPUs = 2. Max memory: 14.563 GB. Platform: Linux.
O^O/ \_/ \    Torch: 2.10.0+cu128. CUDA: 7.5. CUDA Toolkit: 12.8. Triton: 3.6.0
\        /    Bfloat16 = FALSE. FA [Xformers = 0.0.35. FA2 = False]
 "-____-"     Free license: http://github.com/unslothai/unsloth
Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!
Unsloth: Using float16 precision for gemma4 won't work! Using float32.

Test the model with batch inference

If we want to test first the model (before fine-tuning) for the inference on the sentiment analysis dataset, we set then the model for inference:

FastLanguageModel.for_inference(model)

For the inference task on our rather large dataset, we prefer to run batch inference. The procedure to run batch inference for our sentiment analysis task, with a model initialized with Unsloth is shown below:

def predict_batch(df, model, tokenizer, batch_size=8, max_new_tokens=5):
    y_pred = []
    texts = df["text"].tolist()

    for i in tqdm(range(0, len(texts), batch_size)):
        batch = texts[i:i + batch_size]

        messages_batch = [
            [{"role": "user", "content": text}]
            for text in batch
        ]

        prompts = [
            tokenizer.apply_chat_template(
                messages,
                tokenize=False,
                add_generation_prompt=True,
            )
            for messages in messages_batch
        ]

        inputs = tokenizer(
            text=prompts,
            return_tensors="pt",
            padding=True,
            truncation=True,
            max_length=512,
        ).to(model.device)

        input_len = inputs["input_ids"].shape[1]

        with torch.inference_mode():
            outputs = model.generate(
                **inputs,
                max_new_tokens=max_new_tokens,
                do_sample=False,
                use_cache=True,
            )

        decoded = tokenizer.batch_decode(
            outputs[:, input_len:],
            skip_special_tokens=True
        )

        for out in decoded:
            out = out.strip().lower()

            if "positive" in out:
                y_pred.append("positive")
            elif "negative" in out:
                y_pred.append("negative")
            elif "neutral" in out:
                y_pred.append("neutral")
            else:
                y_pred.append("none")

    return y_pred

Prepare model for fine-tuning

We prepare our model for parameter efficient fine tuning using FastLanguageModel.get_peft_model from unsloth.

model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    target_modules = [
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    lora_alpha = 16,
    lora_dropout = 0,
    bias = "none",
    # "unsloth" checkpointing is intended for very long context + lower VRAM
    use_gradient_checkpointing = "unsloth",
    random_state = 3407,
    max_seq_length = max_seq_length,
)

We initialize then the trainer using SFTTrainer . Here we set:

• per_device_train_batch_size = 1 (normal for Gemma 4 and the available memory)
• gradient_accumulation_steps = 4 (well adapted for our task)
• num_training_epochs = 5 (this means, for the size of our training set, and batch size, a total of 1125 steps.

trainer = SFTTrainer(
    model = model,
    train_dataset = train_data,
    tokenizer = tokenizer,
    args = SFTConfig(
        max_seq_length = max_seq_length,
        per_device_train_batch_size = 1,
        gradient_accumulation_steps = 4,
        warmup_steps = 10,
        num_train_epochs = 5,
        logging_steps = 25,
        output_dir = "outputs_gemma4_E2B",
        optim = "adamw_8bit",
        seed = 3407,
        dataset_num_proc = 1,
    ),
)

After running, 3 checkpoints were saved and are present in the Notebook output (checkpoint-500, checkpoint-1000, checkpoint-1125). These do not contain all model weights, but only the LoRA weights. To use the fine-tuned model, we can save these checkpoints and initialize a similar model and just add the LoRA weights.

from unsloth import FastLanguageModel

root_path = "/kaggle/input/notebooks/gpreda/fine-tune-gemma-4-e2b-with-unsloth/"
checkpoint_path = root_path + "outputs_gemma4_E2B/checkpoint-1125"
max_seq_length = 512

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=checkpoint_path,   # point directly to the adapter checkpoint folder
    max_seq_length=max_seq_length,
    load_in_4bit=False,
    load_in_16bit=True,
    full_finetuning=False,
)

FastLanguageModel.for_inference(model)

What I learned

From my first experiments with fine-tuning Gemma 4 on a low-end GPU compute resource, there is no easy way to do it, yet. OOM issues, incompatibility issues make this a little harder. In my experience, Unsloth provides now the most practical path, but the others will follow fast.

Final remarks

Gemma 4 represents a significant step forward in open-weight models, but fine-tuning it today requires careful tooling choices and a willingness to debug low-level issues.

Unsloth proved to be the most reliable way to bridge that gap for the momennt, enabling training even in constrained environments like Kaggle.

While the ecosystem is still evolving, the direction is clear: powerful local fine-tuning is becoming accessible — just not effortless yet.

Check it now

You can fork or download the Notebook and start working on your fine-tuning right now:

[embed]Fine-tune Gemma 4:E2B with Unsloth Explore and run machine learning code with Kaggle Notebooks | Using data from multiple data sourceswww.kaggle.com