Why Your Custom S3 Endpoints Broke in RHOAI 3.4 (And How to Fix It)

Upgrading to Red Hat OpenShift AI (RHOAI) 3.4 in an air-gapped environment? You might run into a frustrating bug: your models suddenly lose the ability to talk to your local object storage fail to load from the storage-initializer.

Here is a quick breakdown of the issue, how to fix it immediately, and why it happened.

The Problem: Stuck in the Cloud

In air-gapped clusters, workloads rely on internal S3-compatible storage (like MinIO, Ceph, or any internal provider).

After upgrading to RHOAI 3.4, models failed to initialize. A quick look at the storage-initializer container logs revealed why:

The system ignored the custom endpoint and tried to hit the default public AWS S3 URI.

Because the cluster had no internet access, the connection timed out, leaving model pods stuck in a crashloopback.

The Quick Fix: Use an Annotation

If your data scientists are blocked, you can bypass the bug manually. Just force Kserve to respect your custom endpoint by adding an annotation directly to your connection secret.

The Workaround

Add the serving.kserve.io/s3-endpoint key to your secret’s metadata:

metadata:
  annotations:
    serving.kserve.io/s3-endpoint: "your-internal-s3-endpoint.local"

Once applied, the initializer will successfully find your internal S3 buckets and pull the models.

The Root Cause: A Missing Environment Variable

Why did this break in 3.4 when it worked perfectly in 3.3? It comes down to a classic upstream code-sync oversight.

• In RHOAI 3.3: The platform looked for the environment variable **AWS_S3_ENDPOINT**. This matched the ODH Dashboard, which automatically writes S3 details using that exact key.
• In RHOAI 3.4: RHOAI synced with upstream Kserve code. Upstream renamed the expected key to **S3_ENDPOINT**.

Because of this mismatch, the Dashboard was still writing AWS_S3_ENDPOINT, but the Kserve Controller was looking for S3_ENDPOINT. When Kserve didn't find its preferred key, it quietly fell back to standard public AWS.

The Long-Term Solution

The RHOAI engineering team has already patched this by configuring the Kserve controller to accept AWS_S3_ENDPOINT as a fallback key.

• The Ultimate Fix: Simply upgrade your RHOAI cluster to a patched version once the official release lands.
• In the meantime: Stick with the annotation workaround to keep your air-gapped models serving smoothly.