Keycloak Upgrade That Broke SAML Logins

Suddenly, a customer ticket landed on our SFDC dashboard, saying that they could not login into our system.

It has been around 12 hours after the deployment of the Keycloak server upgrade. I felt that this might be an unexpected result of this upgrade.

So far all the users could successfully use the system after the upgrade without any kind of issue.

(**Click here** if you are not a Medium member)

Then, what can go wrong with this particular user ?

I extracted the user id from the SFDC ticket and did some investigations on that user account.

This user is coming from a SAML identity provider to login to our application via Keycloak.

But after the upgrade, we have already tested SAML logins before exposing upgraded environment to our customers, we have done all the possible testings including,

• Direct logins ( username/ password)
• SAML logins
• In-house built custom user authenticators
• Custom themes
• In-house built custom cache providers

Every possible scenario was tested, but unfortunately this user can not login into our applications via the SAML identity provider.

We started further investigating the issue. Next finding was that this user is trying to access our service (web application) from their EntraID (Microsoft Azure AD) Identity Provider.

With that I felt like we have missed something.

Even though we tested SAML logins we did not specifically use an EntraID SAML identity provider (IdP). Instead we used another Keycloak instance as the Identity provider to test SAML connections.

When we further investigate this issue on the internet, we found that this login failure is a know one and this is happening when someone tries to login into the Keycloak server via SAML using an EntraID identity provider.

There is a **nice explanation on this issue made by the lead of the Keycloak project [Stian Thorgersen](https://github.com/stianst)**.

Luckily we could fix this login issue by following his instructions and it had happened due to an improper proxy configuration of the Keycloak server.

Issue in Brief

We upgraded Keycloak server from major version 24 to 26. In that we used the same proxy configuration that worked successfully with Keycloak 24 with Keycloak 26 as well. But that proxy configuration is no longer valid with Keycloak 26.

I just wanted to write this post to educate anyone who is doing a Keycloak upgrade about the lessons we learnt from this failed upgrade.

Below are the lessons learnt from this issue.

What we should have done

• Validate each configuration property we have used in the Keycloak configurations against the new version (all Configurations)
• Make the exact identical environment when it comes to testing. (If we used an entraID IDP to test this instead of another Keycloak instance, still we could have caught the issue)

How we mitigated the impact

• We have used the blue-green deployment for this upgrade. Therefore as soon as we got to know of this issue, we routed web traffic back to the blue environment(old keycloak version). So all the users got access to the application again.
• This happened in the user acceptance testing (UAT) environment before doing the production roll out. Therefore the number of affected users were significantly low.

My MANDATORY check list for a Keycloak upgrade. (updated version after the above mentioned failure)

• Go through the **official Keycloak release notes** from the existing Keycloak version to the new version that you are planning to upgrade ( You can get a very good insight about the changes happened between those versions)
• Go through the **Keycloak upgrading guide**. It contains all the breaking changes listed with each version.
• Check for deprecated features or flags that you are currently using.
• Test custom implementations against the new version to be used
• Take a full DB backup before the upgrade
• Test the upgrade in a non-production environment.
• Perform the deployment in blue-green manner.
• Test for all the login flows used in your application. (direct , SAML, etc)
• Check logs for hidden errors
• Monitor the environment at minimum for 24 hours.
• Don’t forget to connect your Keycloak server with an application monitoring tool such as Apache SkyWalking, NewRelic, SumoLogic, ElasticAPM.

This is what have to share on this post. Hope you learnt something new for your next Keycloak upgrade.

Thank you very much

References

• https://www.keycloak.org/docs/latest/upgrading/index.html
• https://www.keycloak.org/docs/latest/release_notes/index.html