I Migrated AWS Services… But a Hidden Redis Cache Broke Production (Here’s How We Found It)

How stale Redis overruled DynamoDB in production

Our services are deployed in AWS and our code is on bitbucket. We planned to migrate our service from bitbucket to GitHub, our single agenda is our code has to be in GitHub, nothing else.

It was so easy, we just use the Git commands to migrate code from bitbucket to GitHub.

But nothing will be easy in DevOps.

Photo by Annie Spratt on Unsplash

The Setup :

Our setup contains ecs services for deployment, cicd as Github action, AWS albs as load balancers, ECR for docker repo.

As part of, bitbucket to Github migration, we migrated several services.

Now the plan is for service-A and service-B.

For these 2 services along with code migration, we also introduced new load balancers to the services instead of old ones because old ones coupled with other services also. For these 2 services I don’t want old ones.

I successfully migrated service-A and service-B from bitbucket to GitHub. No issues seen at the time of migrating.

Also I attached the new albs to the services using iac (terraform), Here also I successfully made the change.

According to the dev team, we need to update the service-A alb url in the dynamoDB , then only service-B can utilize that url from dynamoDB and sent requests to service-A.

So I have updated the service-A url in the dynamoDB table.

The next day we tested both services in staging environment.

Everything seems correct, so I have requested QA team to test service-A, service-B fully, and they tested. All automation testings show good results. I am confident with service-A, service-B.

We kept these 2 services in stage for 3 days. And the next day which Saturday, we planned for production as stage looks good.

I deployed both service-A, service-B with new code and new alb in production stack.

deployment was success.

QA started testing on service-A, got all expected results.

Now comes the service-B, I requested QA to test it. But the test results shows abnormal behaviour.

So I jumped here quickly and checked the logs in cloudwatch.

After careful observation, I noticed one thing.

That is service-B not using the new load balancers urls of service-A, It is using the old urls.

I quickly login to production dynamoDB table, and check the values, But the values here are the new ones only.

I didn’t understand, where it went wrong, from where it is taking those old urls.

Then I reverse engineer the traffic, then I came to see one more thing. That is I totally missed to notice redis presence between service-B and dynamoDB.

This was said by no one, when I ask dev, they say like “oh! I missed to update you about redis cache.” Its fine sometimes mistakes happen, no worries, part of the work.

So I immediately checked what was the impact on production due to this issue. I quickly redeployed the old versions of service-A, service-B. Now production becomes stable.

Now I have to analyse what went wrong and how we missed.

The analysis :

I have go through the internal docs and also I have asked cursor AI to get more details on the service-A, service-B.

DynamoDB table, redis are not part of these services, these are created and maintained in other services. But these 2 services using these infra as point of source.

Service-B has to call to service-A, for that we have to provide service-A’s alb url to service-B. We made service-B to use dynamoDB table for getting the service-A’s alb url.

Everytime reading from the dynamoDB can makes some latency, for this purpose we introduced redis as cache to store the dynamoDB data.

So the overall flow is simply :

Before Migration

          Service-B
              │
              ▼
         Redis Cache (Old URL)
              │
        Cache Hit ✅
              │
              ▼
      Service-A (Old ALB)

Service-B wasn’t reading directly from DynamoDB every time.

After Migration

          Service-B
              │
              ▼
         Redis Cache (Still Old URL)
              │
        Cache Hit ✅
              │
      DynamoDB NOT Queried
              │
              ▼
      Service-A (New ALB)
           ❌ Request Failed

It first checked Redis.

If Redis already had the value, the application never queried DynamoDB.

The same thing happened in my case also.

When I check the redis, I seen a TTL of 12 hours is given in configuration, which means upto 12 hours the data in cache won’t be refreshed and pick from dynamoDB.

This is the reason why service-B still sees the old service-A’s url and routing traffic there.

Why issue not found in STAGE environment?

Yes exactly, If cache makes this problem why stage didn’t get this issue. We have managed same infra, same config in stage exactly similar to production.

But we missed one thing, you remember in above discussions I mentioned that “The next day we tested both services in staging environment.”

The ttl for cache we set is 12 hours, But we requested QA to test stage after 1 day, In this meantime cache got refreshed.

As that is stage, we don’t have real time users we didn’t consider to test in the same day.

The Fix :

Now coming to fix, we have to take these services to production somehow. But we can’t risk 12 hours of downtime.

So I come-up with a suggestion.

Immediately after completing the deployment, I can reboot the cache manually from AWS.

In this way, we got minimal downtime and no calls will be effected.

We implemented the same, we see no Production downtime, no issues, everything is under control.

Service-B
    │
    ▼
Redis (Empty)
    │
Cache Miss
    ▼
DynamoDB
    │
New ALB URL
    ▼
Redis Updated
    │
    ▼
Service-A ✅

Conclusion :

Yes I agree the fix is simple, but the analysis and the root cause for the issue matters alot.

Suppose if we didn’t reboot the redis cache, we may face 12 hours of downtime, which is very critical to handle SLA.

Luckly, the impact is small because we found the issue fastly during QA testing after deployment. End users experienced minimal issue, and I rolled back to previous verion before big production traffic was affected. This bought us enough time to investigate the root cause without violating our deployment safety practices.

Good Devops engineer is not who do the fixes for the issues fastly, he the one who :

1. Found the root cause
2. Analyse why it happened
3. Thinks how to mitigate risk
4. How not to do the same mistake again

Before you go

• Please take a moment to like the post and follow the writer!
• Did you know that over 400,000 developers share what they’re building, learning, and discovering across our platforms every month? Learn how you can contribute here