It’s not often that a zero-day vulnerability causes a network security vendor to urge customers to physically remove and decommission an entire line of affected hardware — as opposed to just applying software updates. But experts say that is exactly what transpired this week with Barracuda Networks, as the company struggled to combat a sprawling malware threat which appears to have undermined its email security appliances in such a fundamental way that they can no longer be safely updated with software fixes.

The Barracuda Email Security Gateway (ESG) 900 appliance.

Campbell, Calif. based Barracuda said it hired incident response firm Mandiant on May 18 after receiving reports about unusual traffic originating from its Email Security Gateway (ESG) devices, which are designed to sit at the edge of an organization’s network and scan all incoming and outgoing email for malware.

On May 19, Barracuda identified that the malicious traffic was taking advantage of a previously unknown vulnerability in its ESG appliances, and on May 20 the company pushed a patch for the flaw to all affected appliances (CVE-2023-2868).

In its security advisory, Barracuda said the vulnerability existed in the Barracuda software component responsible for screening attachments for malware. More alarmingly, the company said it appears attackers first started exploiting the flaw in October 2022.

But on June 6, Barracuda suddenly began urging its ESG customers to wholesale rip out and replace — not patch — affected appliances.

“Impacted ESG appliances must be immediately replaced regardless of patch version level,” the company’s advisory warned. “Barracuda’s recommendation at this time is full replacement of the impacted ESG.”

Rapid7‘s Caitlin Condon called this remarkable turn of events “fairly stunning,” and said there appear to be roughly 11,000 vulnerable ESG devices still connected to the Internet worldwide.

“The pivot from patch to total replacement of affected devices is fairly stunning and implies the malware the threat actors deployed somehow achieves persistence at a low enough level that even wiping the device wouldn’t eradicate attacker access,” Condon wrote.

Barracuda said the malware was identified on a subset of appliances that allowed the attackers persistent backdoor access to the devices, and that evidence of data exfiltration was identified on some systems.

Rapid7 said it has seen no evidence that attackers are using the flaw to move laterally within victim networks. But that may be small consolation for Barracuda customers now coming to terms with the notion that foreign cyberspies probably have been hoovering up all their email for months.

Nicholas Weaver, a researcher at University of California, Berkeley’s International Computer Science Institute (ICSI), said it is likely that the malware was able to corrupt the underlying firmware that powers the ESG devices in some irreparable way.

“One of the goals of malware is to be hard to remove, and this suggests the malware compromised the firmware itself to make it really hard to remove and really stealthy,” Weaver said. “That’s not a ransomware actor, that’s a state actor. Why? Because a ransomware actor doesn’t care about that level of access. They don’t need it. If they’re going for data extortion, it’s more like a smash-and-grab. If they’re going for data ransoming, they’re encrypting the data itself — not the machines.”

In addition to replacing devices, Barracuda says ESG customers should also rotate any credentials connected to the appliance(s), and check for signs of compromise dating back to at least October 2022 using the network and endpoint indicators the company has released publicly.