Shifting left your debug thought process

Reddit

Introduction

The following article is a little different from what I usually post, it is not about anything technical, not about kubernetes or even security related. This time I’d like to share a unique debugging session I took part of, and use it as a case study, and how it make me thing differently.

Case study

While scrolling through Reddit in my free time, I have encountered a ‘help needed’ thread. In the thread, a reddit user, whom we will name Jimmy, had asked for help in debugging what went wrong in his web-application deployment, fronted by nginx, using AWS Elastic Beanstalk service.

Jimmy had set up everything needed to run his application successfully, and deployed it in the us-west-1 region in AWS. Luckily, Jimmy had done a pretty good job and not a lot of tweaking were needed.

Jimmy had taken his configurations and deployed the same architectural pattern in another region, this time, cn-north-1, which resides in Beijing, China.

Do you think it worked?

Well, thankfully no, otherwise I would have had nothing to write about 🫢

Jimmy kept getting “Connection had timed out” error message whenever he was trying to reach the EBS (Elastic Beanstalk) domain. However, he was able to reach the application from within the underlying EC2 instance using curl 127.0.0.1 .

Me being a challenge-enthusiast, and a great friend, I had reached out trying to help tackle this issue. My experience with “Connection timed out” issues lead me to understand Jimmy is not able to reach the nginx proxy at all, so I asked Jimmy to check the nginx logs to verify that. Indeed, no entries were logged.

So even though I’m a a firm believer that “DNS IS ALWAYS THE PROBLEM”, I tought we should put it up to the test. Perhaps maybe the domains is not pointing correctly at the instance — this time I’ve asked Jimmy to try reaching the web-application using the EC2 instance public IP. Unfortunately, same results appeared.

Don’t give up Jimmy, it ain’t over yet!

At this point I had 2 Hypotheses:

1. The services might not binding to all of the Ethernet interfaces, may be only publish across the loop interface only — Due to the errors given, I assumed not information is not flowing through the ports (http, https & 8080 ports).
2. China is blocking traffic originating from the US — I know, it’s bizzare, but due to the eternal feud between the two, there might be geo-location restriction set one way or another, obsecured as a “security” measure.

u/koudingspawn to the rescue!

Right about now another user jumped in pointing out the by default ports 80 (HTTP), 443 (HTTPS) and 8080 are blocked in China. This is due to legal reasons, and a request targeted at the AWS support is needed in order to allow the uses of them. Also, an ICP is needed for websites in China.

The parable of the story

Shift left is the practice of moving testing, quality, and performance evaluation early in the development process, often before any code is written. Shift left testing helps teams anticipate changes that arise during the development process that can affect performance or other.

• Credit

Migration / replication is never easy, and even the perfect configurations, which proved to be working as needed, might fails or encounter unknowns issues. I had a decent train of thought in this debugging session, but I could always improve 😄

and most important, you should always shift left, meaning you should discover and handle bugs / problems / misconfigurations early on in the process. To put this into practice in this article, this should have been discovered in the architecture stage, or in the prior research stage.