It’s not often that a zero-day vulnerability causes a network security vendor to urge customers to physically remove and decommission an entire line of affected hardware — as opposed to just applying software updates. But experts say that is exactly what transpired this week with Barracuda Networks, as the company struggled to combat a sprawling malware threat which appears to have undermined its email security appliances in such a fundamental way that they can no longer be safely updated with software fixes.

The Barracuda Email Security Gateway (ESG) 900 appliance.

Campbell, Calif. based Barracuda said it hired incident response firm Mandiant on May 18 after receiving reports about unusual traffic originating from its Email Security Gateway (ESG) devices, which are designed to sit at the edge of an organization’s network and scan all incoming and outgoing email for malware.

On May 19, Barracuda identified that the malicious traffic was taking advantage of a previously unknown vulnerability in its ESG appliances, and on May 20 the company pushed a patch for the flaw to all affected appliances (CVE-2023-2868).

In its security advisory, Barracuda said the vulnerability existed in the Barracuda software component responsible for screening attachments for malware. More alarmingly, the company said it appears attackers first started exploiting the flaw in October 2022.

But on June 6, Barracuda suddenly began urging its ESG customers to wholesale rip out and replace — not patch — affected appliances.

“Impacted ESG appliances must be immediately replaced regardless of patch version level,” the company’s advisory warned. “Barracuda’s recommendation at this time is full replacement of the impacted ESG.”

In a statement, Barracuda said it will be providing the replacement product to impacted customers at no cost, and that not all ESG appliances were compromised.

“No other Barracuda product, including our SaaS email solutions, were impacted by this vulnerability,” the company said. “If an ESG appliance is displaying a notification in the User Interface, the ESG appliance had indicators of compromise. If no notification is displayed, we have no reason to believe that the appliance has been compromised at this time.”

Nevertheless, the statement says that “out of an abundance of caution and in furtherance of our containment strategy, we recommend impacted customers replace their compromised appliance.”

“As of June 8, 2023, approximately 5% of active ESG appliances worldwide have shown any evidence of known indicators of compromise due to the vulnerability,” the statement continues. “Despite deployment of additional patches based on known IOCs, we continue to see evidence of ongoing malware activity on a subset of the compromised appliances. Therefore, we would like customers to replace any compromised appliance with a new unaffected device.”

Rapid7‘s Caitlin Condon called this remarkable turn of events “fairly stunning,” and said there appear to be roughly 11,000 vulnerable ESG devices still connected to the Internet worldwide.

“The pivot from patch to total replacement of affected devices is fairly stunning and implies the malware the threat actors deployed somehow achieves persistence at a low enough level that even wiping the device wouldn’t eradicate attacker access,” Condon wrote.

Barracuda said the malware was identified on a subset of appliances that allowed the attackers persistent backdoor access to the devices, and that evidence of data exfiltration was identified on some systems.

Rapid7 said it has seen no evidence that attackers are using the flaw to move laterally within victim networks. But that may be small consolation for Barracuda customers now coming to terms with the notion that foreign cyberspies probably have been hoovering up all their email for months.

Nicholas Weaver, a researcher at University of California, Berkeley’s International Computer Science Institute (ICSI), said it is likely that the malware was able to corrupt the underlying firmware that powers the ESG devices in some irreparable way.

“One of the goals of malware is to be hard to remove, and this suggests the malware compromised the firmware itself to make it really hard to remove and really stealthy,” Weaver said. “That’s not a ransomware actor, that’s a state actor. Why? Because a ransomware actor doesn’t care about that level of access. They don’t need it. If they’re going for data extortion, it’s more like a smash-and-grab. If they’re going for data ransoming, they’re encrypting the data itself — not the machines.”

In addition to replacing devices, Barracuda says ESG customers should also rotate any credentials connected to the appliance(s), and check for signs of compromise dating back to at least October 2022 using the network and endpoint indicators the company has released publicly.

Update, June 9, 11:55 a.m. ET: Barracuda has issued an updated statement about the incident, portions of which are now excerpted above.