Another “failure-to-patch” breach, this time of an (anonymous) federal agency

It’s a story that is by now familiar — too familiar. An organization gets breached by cyberattackers who exploit a vulnerability in a component of its software supply chain — a vulnerability for which a patch had been available for years.

What makes this version of the story a bit more interesting is that the organization is a federal agency. A March 15 joint advisory from the Cybersecurity and Infrastructure Security Agency (CISA), the FBI, and the Multi-State Information Sharing and Analysis Center reported that a federal civilian executive branch (FCEB) agency was breached.

According to the advisory, exploit activities by one group likely began in August 2021 and by another in August 2022. CISA and the other agencies had “identified the presence of indicators of compromise” from November 2022 through early January this year.

The attackers exploited a code execution vulnerability in a developer tool known as the Telerik user interface (UI), which bundles more than 100 UI components that can help developers create custom web applications faster. The Common Vulnerability Scoring System gave it a Common Vulnerabilities and Exposures (CVE) severity rating of 9.8 out of 10 — among the worst of the worst.

Code execution vulnerabilities, which are among the most common application vulnerabilities, allow an attacker to run malicious code on the system hosting the vulnerable application, and then disrupt its operations and/or the operations of other applications on the system.

The Telerik UI is sold by a company called Progress, which released a patch for the vulnerability in late 2019.

The reason the “federal agency” part of the case makes it a bit more interesting is that the leader of the federal government has for years been preaching about the need for better software supply chain security. President Joe Biden made it a major element of his May 2021 Executive Order (EO) on Improving the Nation’s Cybersecurity, and it also figures prominently in his National Cybersecurity Strategy, released earlier this month.

The strategy document even calls for government regulation — for organizations to be held liable for damages resulting from the failure of organizations to “take reasonable precautions to secure their software.” And failure to apply a patch that had been available for more than three years would clearly fail the “reasonable precautions” test.

Stop requesting, start mandating

Of course, neither the EO nor the strategy have been fully implemented yet. The proposed liability provision may never be, since that would require congressional legislation. Still, the president has been using the bully pulpit to call for government to stop requesting and start mandating much better software supply chain security, especially by the private sector organizations that run most of the nation’s critical infrastructure.

Which is a very good thing. There is little to no debate among cybersecurity experts that everything from the power grid to transportation infrastructure, water and sewer, banking, fuel, medical services, and food supply can be taken down as effectively by cyberattacks as by bombs and missiles.

Ransomware attacks over the past couple of years that led to the shutdown for about a week of the Colonial Pipeline, the main fuel supply line to the U.S. East Coast, and on JBS Foods, the world’s largest meat supplier, made that clear.

And those were just about money, not a nation state trying to weaken an adversary. That kind of threat was demonstrated in December 2016 when hackers, allegedly from Russian intelligence, took down a portion of the energy grid in Ukraine.

The unnamed FCEB agency is no outlier. Thousands of organizations have failed to apply available patches to vulnerabilities in their software supply chains. The annual Open Source Security and Risk Analysis (OSSRA) report by Synopsys, in an analysis of more than 1,700 codebases, found that 89% had open source software components that were more than four years out-of-date and 48% included high-risk vulnerabilities. That has generated widespread support for government mandates to force better security in a sector where market pressure won’t do it.

But government services are critical too, which makes the security of government agency software is just as important.

And that highlights another interesting thing. The advisory, while it covers the “what” and the “how” of the event, leaves out the “who.” It doesn’t name the attackers, calling them “threat actors,” perhaps because they haven’t yet been identified, but it also doesn’t identify the agency that was breached, nor does it say what, if any, data or other information was compromised.

Info sharing gaps

That appears to run counter to both the Biden EO and the strategy document, which call for more threat and breach information sharing between the public and private sectors.

Private sector organizations and privacy advocates have complained for years that the government version of information sharing is a “one-way street” in which they are expected to share while government doesn’t. This makes it look like they have a point.

CISA had not responded to a request for comment on that when this was posted.

The advisory also doesn’t report why the vulnerability wasn’t patched.

An expert with a major think tank who declined to be identified said it was likely because “the agency’s vulnerability scanners missed it. The agency definitely wasn’t doing the right thing, but maybe not because [its leaders] didn’t care but because they didn’t even know this component was there and needed a patch.”

Indeed, that’s probable — one of the chief reasons organizations end up getting breached is that they don’t know what they’re using. As the saying in cybersecurity goes, “you can’t protect what you don’t know you have.”

But that exact problem is one of the things the Biden EO addresses — that organizations should keep track of every component of the software they’re using. And it’s not as though the federal government is not aware of the weaknesses of vulnerability scanners. The CISA advisory noted that “vulnerability scanners may have limitations in detecting vulnerabilities, such as only being able to identify Windows Installer-installed applications, which was the case with this agency’s vulnerability scanner.”

“The Telerik UI software was installed via a continuous integration and continuous delivery pipeline rather than the Windows Installer. This highlights the importance of using a comprehensive approach for vulnerability scanning that considers all potential installation methods and file paths.”

Complex but possible

It’s true that solving this problem is complicated. The software supply chain is much more labyrinthine than it might first appear. A single application may have only a dozen or so “declared” components. But each of them may depend on other components, called dependencies, to function properly. And those other components may have dependencies of their own, going multiple levels deep. There is no way to track that manually.

Beyond that, installing updates in complicated systems can be dicier than simply tapping a button on your smartphone to update one of your mobile apps. A comment from a reader on a story about the breach in Ars Technica noted that in their organization, “more than once we were told NOT to proceed with a needed update because said update (containing security fixes) also contained UI/frontend changes that broke interfaces or made some other unwanted change to the behavior of UI […] someone holding the money bag would tell us ‘STOP — it’s too expensive’.”

The CISA advisory doesn’t say if that was the case at the FCEB.

But complicated doesn’t mean impossible. CISA, other agencies, and the president, along with a host of cybersecurity companies, experts, and reports like the OSSRA have for years told organizations that automated technology is available to help organizations keep track of their software and keep it up-to-date.

A software Bill of Materials (SBOM), which can be created with the help of an automated software composition analysis(SCA) tool, will tell organizations what software components they’re using and also if any of them have known vulnerabilities.

The Biden EO even calls for federal agencies to be banned from purchasing any software product that doesn’t include an SBOM — although that directive is still in process nearly two years later.

Beyond that advice, the federal National Institute of Standards and Technology issued a special publication titled Cybersecurity Supply Chain Risk Management Practices for Systems and Organizations in May 2022 that goes into exhaustive detail (more than 300 pages) on how to mitigate the risks of supply chain attacks. The SBOM acronym shows up in that document 27 times.

Finally, the CISA advisory includes “actions to take today to mitigate malicious cyber activity.” First on the list is for organizations (and agencies) to “implement a patch management solution to ensure compliance with the latest security patches.”

Perhaps, if the president wants to set an example, he could impose some consequences on the unnamed agency for failing to take the “reasonable” security precautions he wants to mandate for the private sector. Given that the agency is under his authority, he wouldn’t need legislation to do it.

Meanwhile, the think tank expert said that “in some sense this story illustrates how essential it is for organizations to have processes for the whole of the software and device management lifecycle — just slapping on an out-of-the-box vulnerability scan without understanding whether it matches how your organization does stuff is not enough.”