CISA

VMware Flaw a Vector in SolarWinds Breach?

U.S. government cybersecurity agencies warned this week that the attackers behind the widespread hacking spree stemming from the compromise at network software firm SolarWinds used weaknesses in other, non-SolarWinds products to attack high-value targets. According to sources, among those was a flaw in software virtualization platform VMware, which the U.S. National Security Agency (NSA) warned on Dec. 7 was being used by Russian hackers to impersonate authorized users on victim networks.

On Dec. 7, 2020, the NSA said “Russian state-sponsored malicious cyber actors are exploiting a vulnerability in VMware Access and VMware Identity Manager products, allowing the actors access to protected data and abusing federated authentication.”

VMware released a software update to plug the security hole (CVE-2020-4006) on Dec. 3, and said it learned about the flaw from the NSA.

The NSA advisory (PDF) came less than 24 hours before cyber incident response firm FireEye said it discovered attackers had broken into its networks and stolen more than 300 proprietary software tools the company developed to help customers secure their networks.

On Dec. 13, FireEye disclosed that the incident was the result of the SolarWinds compromise, which involved malicious code being surreptitiously inserted into updates shipped by SolarWinds for users of its Orion network management software as far back as March 2020.

In its advisory on the VMware vulnerability, the NSA urged patching it “as soon as possible,” specifically encouraging the National Security System, Department of Defense, and defense contractors to make doing so a high priority.

The NSA said that in order to exploit this particular flaw, hackers would already need to have access to a vulnerable VMware device’s management interface — i.e., they would need to be on the target’s internal network (provided the vulnerable VMware interface was not accessible from the Internet). However, the SolarWinds compromise would have provided that internal access nicely.

In response to questions from KrebsOnSecurity, VMware said it has “received no notification or indication that the CVE 2020-4006 was used in conjunction with the SolarWinds supply chain compromise.”

VMware added that while some of its own networks used the vulnerable SolarWinds Orion software, an investigation has so far revealed no evidence of exploitation.

“While we have identified limited instances of the vulnerable SolarWinds Orion software in our environment, our own internal investigation has not revealed any indication of exploitation,” the company said in a statement. “This has also been confirmed by SolarWinds own investigations to date.”

On Dec. 17, DHS’s Cybersecurity and Infrastructure Security Agency (CISA) released a sobering alert on the SolarWinds attack, noting that CISA had evidence of additional access vectors other than the SolarWinds Orion platform.

CISA’s advisory specifically noted that “one of the principal ways the adversary is accomplishing this objective is by compromising the Security Assertion Markup Language (SAML) signing certificate using their escalated Active Directory privileges. Once this is accomplished, the adversary creates unauthorized but valid tokens and presents them to services that trust SAML tokens from the environment. These tokens can then be used to access resources in hosted environments, such as email, for data exfiltration via authorized application programming interfaces (APIs).”

Indeed, the NSA’s Dec. 7 advisory said the hacking activity it saw involving the VMware vulnerability “led to the installation of a web shell and follow-on malicious activity where credentials in the form of SAML authentication assertions were generated and sent to Microsoft Active Directory Federation Services (ADFS), which in turn granted the actors access to protected data.”

Also on Dec. 17, the NSA released a far more detailed advisory explaining how it has seen the VMware vulnerability being used to forge SAML tokens, this time specifically referencing the SolarWinds compromise.

Asked about the potential connection, the NSA said only that “if malicious cyber actors gain initial access to networks through the SolarWinds compromise, the TTPs [tactics, techniques and procedures] noted in our December 17 advisory may be used to forge credentials and maintain persistent access.”

“Our guidance in this advisory helps detect and mitigate against this, no matter the initial access method,” the NSA said.

CISA’s analysis suggested the crooks behind the SolarWinds intrusion were heavily focused on impersonating trusted personnel on targeted networks, and that they’d devised clever ways to bypass multi-factor authentication (MFA) systems protecting networks they targeted.

The bulletin references research released earlier this week by security firm Volexity, which described encountering the same attackers using a novel technique to bypass MFA protections provided by Duo for Microsoft Outlook Web App (OWA) users.

Duo’s parent Cisco Systems Inc. responded that the attack described by Volexity didn’t target any specific vulnerability in its products. As Ars Technica explained, the bypass involving Duo’s protections could have just as easily involved any of Duo’s competitors.

“MFA threat modeling generally doesn’t include a complete system compromise of an OWA server,” Ars’ Dan Goodin wrote. “The level of access the hacker achieved was enough to neuter just about any defense.”

Several media outlets, including The New York Times and The Washington Post, have cited anonymous government sources saying the group behind the SolarWinds hacks was known as APT29 or “Cozy Bear,” an advanced threat group believed to be part of the Russian Federal Security Service (FSB).

SolarWinds has said almost 18,000 customers may have received the backdoored Orion software updates. So far, only a handful of customers targeted by the suspected Russian hackers behind the SolarWinds compromise have been made public — including the U.S. Commerce, Energy and Treasury departments, and the DHS.

No doubt we will hear about new victims in the public and private sector in the coming days and weeks. In the meantime, thousands of organizations are facing incredibly costly, disruptive and time-intensive work in determining whether they were compromised and if so what to do about it.

The CISA advisory notes the attackers behind the SolarWinds compromises targeted key personnel at victim firms — including cyber incident response staff, and IT email accounts. The warning suggests organizations that suspect they were victims should assume their email communications and internal network traffic are compromised, and rely upon or build out-of-band systems for discussing internally how they will proceed to clean up the mess.

“If the adversary has compromised administrative level credentials in an environment—or if organizations identify SAML abuse in the environment, simply mitigating individual issues, systems, servers, or specific user accounts will likely not lead to the adversary’s removal from the network,” CISA warned. “In such cases, organizations should consider the entire identity trust store as compromised. In the event of a total identity compromise, a full reconstitution of identity and trust services is required to successfully remediate. In this reconstitution, it bears repeating that this threat actor is among the most capable, and in many cases, a full rebuild of the environment is the safest action.”

SolarWinds Hack Could Affect 18K Customers

The still-unfolding breach at network management software firm SolarWinds may have resulted in malicious code being pushed to nearly 18,000 customers, the company said in a legal filing on Monday. Meanwhile, Microsoft should soon have some idea which and how many SolarWinds customers were affected, as it recently took possession of a key domain name used by the intruders to control infected systems.

On Dec. 13, SolarWinds acknowledged that hackers had inserted malware into a service that provided software updates for its Orion platform, a suite of products broadly used across the U.S. federal government and Fortune 500 firms to monitor the health of their IT networks.

In a Dec. 14 filing with the U.S. Securities and Exchange Commission (SEC), SolarWinds said roughly 33,000 of its more than 300,000 customers were Orion customers, and that fewer than 18,000 customers may have had an installation of the Orion product that contained the malicious code. SolarWinds said the intrusion also compromised its Microsoft Office 365 accounts.

The initial breach disclosure from SolarWinds came five days after cybersecurity incident response firm FireEye announced it had suffered an intrusion that resulted in the theft of some 300 proprietary software tools the company provides to clients to help secure their IT operations.

On Dec. 13, FireEye published a detailed writeup on the malware infrastructure used in the SolarWinds compromise, presenting evidence that the Orion software was first compromised back in March 2020. FireEye didn’t explicitly say its own intrusion was the result of the SolarWinds hack, but the company confirmed as much to KrebsOnSecurity earlier today.

Also on Dec. 13, news broke that the SolarWinds hack resulted in attackers reading the email communications at the U.S. Treasury and Commerce departments.

On Dec. 14, Reuters reported the SolarWinds intrusion also had been used to infiltrate computer networks at the U.S. Department of Homeland Security (DHS). That disclosure came less than 24 hours after DHS’s Cybersecurity and Infrastructure Security Agency (CISA) took the unusual step of issuing an emergency directive ordering all federal agencies to immediately disconnect the affected Orion products from their networks.

ANALYSIS

Security experts have been speculating as to the extent of the damage from the SolarWinds hack, combing through details in the FireEye analysis and elsewhere for clues about how many other organizations may have been hit.

And it seems that Microsoft may now be in perhaps the best position to take stock of the carnage. That’s because sometime on Dec. 14, the software giant took control over a key domain name — avsvmcloud[.]com — that was used by the SolarWinds hackers to communicate with systems compromised by the backdoored Orion product updates.



Armed with that access, Microsoft should be able to tell which organizations have IT systems that are still trying to ping the malicious domain. However, because many Internet service providers and affected companies are already blocking systems from accessing that malicious control domain or have disconnected the vulnerable Orion services, Microsoft’s visibility may be somewhat limited.

Microsoft has a long history of working with federal investigators and the U.S. courts to seize control over domains involved in global malware menaces, particularly when those sites are being used primarily to attack Microsoft Windows customers.

Microsoft dodged direct questions about its visibility into the malware control domain, suggesting those queries would be better put to FireEye or GoDaddy (the current domain registrar for the malware control server). But in a response on Twitter, Microsoft spokesperson Jeff Jones seemed to confirm that control of the malicious domain had changed hands.

“We worked closely with FireEye, Microsoft and others to help keep the internet safe and secure,” GoDaddy said in a written statement. “Due to an ongoing investigation and our customer privacy policy, we can’t comment further at this time.”

FireEye declined to answer questions about exactly when it learned of its own intrusion via the Orion compromise, or approximately when attackers first started offloading sensitive tools from FireEye’s network. But the question is an interesting one because its answer may speak to the motivations and priorities of the hackers.

Based on the timeline known so far, the perpetrators of this elaborate hack would have had a fairly good idea back in March which of SolarWinds’ 18,000 Orion customers were worth targeting, and perhaps even in what order.

Alan Paller, director of research for the SANS Institute, a security education and training company based in Maryland, said the attackers likely chose to prioritize their targets based on some calculation of risk versus reward.

Paller said the bad guys probably sought to balance the perceived strategic value of compromising each target with the relative likelihood that exploiting them might result in the entire operation being found out and dismantled.

“The way this probably played out is the guy running the cybercrime team asked his people to build a spreadsheet where they ranked targets by the value of what they could get from each victim,” Paller said. “And then next to that they likely put a score for how good the malware hunters are at the targets, and said let’s first go after the highest priority ones that have a hunter score of less than a certain amount.”

The breach at SolarWinds could well turn into an existential event for the company, depending on how customers react and how SolarWinds is able to weather the lawsuits that will almost certainly ensue.

“The lawsuits are coming, and I hope they have a good general counsel,” said James Lewis, senior vice president at the Center for Strategic and International Studies. “Now that the government is telling people to turn off [the SolarWinds] software, the question is will anyone turn it back on?”

According to its SEC filing, total revenue from the Orion products across all customers — including those who may have had an installation of the Orion products that contained the malicious update — was approximately $343 million, or roughly 45 percent of the firm’s total revenue. SolarWinds’ stock price has fallen 25 percent since news of the breach first broke.

Some of the legal and regulatory fallout may hinge on what SolarWinds knew or should have known about the incident, when, and how it responded. For example, Vinoth Kumar, a cybersecurity “bug hunter” who has earned cash bounties and recognition from multiple companies for reporting security flaws in their products and services, posted on Twitter that he notified SolarWinds in November 2019 that the company’s software download website was protected by a simple password that was published in the clear on SolarWinds’ code repository at Github.

Andrew Morris, founder of the security firm GreyNoise Intelligence, on said that as of Tuesday evening SolarWinds still hadn’t removed the compromised Orion software updates from its distribution server.

Another open question is how or whether the incoming U.S. Congress and presidential administration will react to this apparently broad cybersecurity event. CSIS’s Lewis says he doubts lawmakers will be able to agree on any legislative response, but he said it’s likely the Biden administration will do something.

“It will be a good new focus for DHS, and the administration can issue an executive order that says federal agencies with regulatory authority need to manage these things better,” Lewis said. “But whoever did this couldn’t have picked a better time to cause a problem, because their timing almost guarantees a fumbled U.S. response.”

Trump Fires Security Chief Christopher Krebs

President Trump on Tuesday fired his top election security official Christopher Krebs (no relation). The dismissal came via Twitter two weeks to the day after Trump lost an election he baselessly claims was stolen by widespread voting fraud.

Chris Krebs. Image: CISA.

Krebs, 43, is a former Microsoft executive appointed by Trump to head the Cybersecurity and Infrastructure Security Agency (CISA), a division of the U.S. Department of Homeland Security. As part of that role, Krebs organized federal and state efforts to improve election security, and to dispel disinformation about the integrity of the voting process.

Krebs’ dismissal was hardly unexpected. Last week, in the face of repeated statements by Trump that the president was robbed of re-election by buggy voting machines and millions of fraudulently cast ballots, Krebs’ agency rejected the claims as “unfounded,” asserting that “the November 3rd election was the most secure in American history.”

In a statement on Nov. 12, CISA declared “there is no evidence that any voting system deleted or lost votes, changed votes, or was in any way compromised.”

But in a tweet Tuesday evening, Trump called that assessment “highly inaccurate,” alleging there were “massive improprieties and fraud — including dead people voting, Poll watchers not allowed into polling locations, ‘glitches’ in the voting machines that changed votes from Trump to Biden, late voting, and many more.”

Twitter, as it has done with a remarkable number of the president’s tweets lately, flagged the statements as disputed.

By most accounts, Krebs was one of the more competent and transparent leaders in the Trump administration. But that same transparency may have cost him his job: Krebs’ agency earlier this year launched “Rumor Control,” a blog that sought to address many of the conspiracy theories the president has perpetuated in recent days.

Sen. Richard Burr, a Republican from North Carolina, said Krebs had done “a remarkable job during a challenging time,” and that the “creative and innovative campaign CISA developed to promote cybersecurity should serve as a model for other government agencies.”

Sen. Angus King, an Independent from Maine and co-chair of a commission to improve the nation’s cyber defense posture, called Krebs “an incredibly bright, high-performing, and dedicated public servant who has helped build up new cyber capabilities in the face of swiftly-evolving dangers.”

“By firing Mr. Krebs for simply doing his job, President Trump is inflicting severe damage on all Americans – who rely on CISA’s defenses, even if they don’t know it,” King said in a written statement. “If there’s any silver lining in this unjust decision, it’s this: I hope that President-elect Biden will recognize Chris’s contributions, and consult with him as the Biden administration charts the future of this critically important agency.”

KrebsOnSecurity has received more than a few messages these past two weeks from readers who wondered why the much-anticipated threat from Russian or other state-sponsored hackers never appeared to materialize in this election cycle.

That seems a bit like asking why the year 2000 came to pass with very few meaningful disruptions from the Y2K computer date rollover problem. After all, in advance of the new millennium, the federal government organized a series of task forces that helped coordinate readiness for the changeover, and to minimize the impact of any disruptions.

But the question also ignores a key goal of previous foreign election interference attempts leading up to the 2016 U.S. presidential and 2018 mid-term elections. Namely, to sow fear, uncertainty, doubt, distrust and animosity among the electorate about the democratic process and its outcomes.

To that end, it’s difficult to see how anyone has done more to advance that agenda than President Trump himself, who has yet to concede the race and continues to challenge the result in state courts and in his public statements.