MDRs: Security Professionals or Configuration Managers?

Photo by Ofspace LLC on Unsplash

Whether it’s the firewall being open because the CFO needs remote access from Vietnam, or a client’s files having not only SSNs in the file names but the employee’s names as well (yes, I really had to tell someone making 3x my salary that that’s not okay), every security professional has had that moment that made them lose just a little bit of faith in the world.

What MDR analysts actually spend their time on

When you’re onboarding 10 companies a month, on top of managing the existing ones, it gets hard to keep up with everything. Generic runbooks, 20 different RMM and backup softwares across the board, and client escalation points who know less about the environment than you do. Managed Detection and Response analysts are hired primarily to detect and respond, however the reality is that most of their time is spent on managing.

A significant portion of their day-to-day work is spent managing the fallout from environments that were never properly maintained to begin with. The detection and response quality degrade as a result, but the blame is almost always placed on the analysts, not the ones who made the poor configurations in the first place.

From my experience, around 40–50% of alerts are caused by these poor configurations. A threat actor scanning open ports that we told the client to close 2 weeks ago. Old ransomware files being detected because the IT manager won’t remove them from the production environment.

Of the remaining alerts, most of it’s just RMMs and backup software’s doing their thing. After 2 years of looking at alerts, I feel like I know more about NinjaOne and Datto CentraStage than I do about myself. There is nothing more disheartening than thinking you finally got some action, but it’s just a legitimate Connectwise process doing some funky stuff in the background.

The exclusion problem

The answer to this problem seems simple: just make the appropriate exclusions. While that sounds like the obvious thing to do in theory, what happens when a threat actor gets a hold of your RMM? Now instead of 6 alerts that clearly represent an attack chain, you have 1 or 2 that look exactly like the ConnectWise process you have been clearing for three months. The breach doesn’t look like a breach anymore. It looks like another Tuesday.

The real solution doesn’t always exist, because you generally have 2 choices: either make the exclusion very narrow, and risk the same RMM to continue firing every day, or make the exclusion more broad, and you risk visibility.

This is where alert fatigue tips into something worse. After months of clearing the same RMM alerts, you stop opening them the same way you used to. You clear it before you finish reading. That’s not laziness, it’s any SOC manager’s biggest nightmare: alert desensitization. It’s what happens when a pattern gets reinforced hundreds of times, and it just happens to be exactly the condition a threat actor with RMM access is counting on.

How client behavior degrades detection quality over time

When you onboard a client, you are going to have to learn their environment, and a big part of that process is in the form of calling them over things that are probably benign, but you just have to call them to make sure. This can get very old very fast, however there’s no avoiding it. What might be legitimate activity in one environment might be a telltale sign of a breach in another.

The problem here is that not every client understands this, and not all of them think about how they aren’t your only client.

Let’s say you just finished dealing with an incident caused by a breached RMM. Now you get an alert from a newer client that is detecting almost identical activity. You call them. You get yelled at for “not knowing basic RMM principles”. Now you never want to talk to them again.

Whether it’s one client, or a pattern across multiple of them, when you are constantly told everything is benign, after even just 3–4 months, it’s hard to stay engaged and alert regarding that one domain of activity. It’s hard to want to triage every alert caused by an RMM. You know that there’s a non-zero chance of a breach, but after months of clients telling you that’s impossible, it gets hard.

This plays into the desensitization problem we talked about earlier. At some point, the rational response to being wrong every time is to stop treating it as a real question. It’s not a character flaw on the analyst, it’s just how pattern recognition can work against you.

The leadership layer

Anyone who has ever been tied to a MDR or SOC environment know the real risks that come with alert fatigue and desensitization. Responding to a detection too late because of too many alerts leading to a higher MTTR. The normalization of genuine signals. Undocumented risk threshold shifts. All of these problems will eventually lead to the one that is hardest to overcome: analyst burnout and turnover.

The root of these problems come from a place a lot of people don’t want to look at. Leadership, from all the way up top.

Everyone knows that there are too many alerts. Everyone knows that the analysts are getting belittled every day by clients for just doing their jobs. The analysts know, leadership knows, and more importantly, they each know that the other knows. The problem isn’t a leadership problem, it’s the business model.

At the end of the day, what is the reason any given business exists? To make money. The reason that these problems exist and no one does anything about them is because the real incentive is retention, not detection quality. Detection quality is just a sales tool used to gain new clients and increase that retention.

The CISO’s jobs is to turn security into dollars for the CEO. The security director’s job is to execute the plans made by the CISO to increase those dollars. The MDR/SOC manager’s job is to make sure the analysts are doing their jobs correctly and efficiently, prioritizing the customer’s experience. Who’s left to make sure that those analysts don’t burn out, become desensitized, and either let a breach through, or best case scenario, they quit and now you have to train a new analyst. Nobody in that chain is accountable for what happens inside the analyst’s head after eighteen months of being the buffer between a client who does not care and a leadership team that has decided that’s acceptable as long as the check clears.

The solution

There is no clean solution here, and anyone trying to sell you one is proving the point. This is not a solution that can be sold, it’s something that has to be believed in; practiced, not preached.

The exclusions will never be perfect. The clients will never truly, fully understand why you are calling them at 2am. The alerts will keep coming, and at some point, every analyst finds their own way of coping with the volume. And not all of those coping mechanisms are good for detection quality.

If you are an analyst reading this, you already know everything I just said. You’ve probably already said it yourself in the late afternoon shift after HR left the building. The frustrating reality is not only that you aren’t the problem, but you aren’t in a position to fix the actual issues. You’re doing exactly what a rational person does when an environment consistently tells them that their judgement doesn’t matter, even though your judgement is the exact thing you were hired to provide.

If you’re on the client side, the honest thing to say is this: your MDR is only as good as the environment you hand them. Every configuration you have been meaning to fix, every exclusion that got made because someone complained, every alert your team said was benign without actually checking; that is not staying on your side of the fence. It’s sitting in someone else’s queue, training pattern recognition, and quietly lowering the bar for what is treated as worth investigating. You are not a passive recipient of your MDR service. You are part of the product as much as they are.

If your toes were stepped on, this article was meant for you, and you know there is plenty of truth behind what I said. The analysts doing the work are not burned out because they are weak, or lazy. They are burnt out because they actually care too much, and caring inside a system that was not designed to reward, or even recognize it, is exhausting. That is something worth acknowledging, because no runbook is going to fix it for you.