The Coming Storm

VMware Flaw a Vector in SolarWinds Breach?

U.S. government cybersecurity agencies warned this week that the attackers behind the widespread hacking spree stemming from the compromise at network software firm SolarWinds used weaknesses in other, non-SolarWinds products to attack high-value targets. According to sources, among those was a flaw in software virtualization platform VMware, which the U.S. National Security Agency (NSA) warned on Dec. 7 was being used by Russian hackers to impersonate authorized users on victim networks.

On Dec. 7, 2020, the NSA said “Russian state-sponsored malicious cyber actors are exploiting a vulnerability in VMware Access and VMware Identity Manager products, allowing the actors access to protected data and abusing federated authentication.”

VMware released a software update to plug the security hole (CVE-2020-4006) on Dec. 3, and said it learned about the flaw from the NSA.

The NSA advisory (PDF) came less than 24 hours before cyber incident response firm FireEye said it discovered attackers had broken into its networks and stolen more than 300 proprietary software tools the company developed to help customers secure their networks.

On Dec. 13, FireEye disclosed that the incident was the result of the SolarWinds compromise, which involved malicious code being surreptitiously inserted into updates shipped by SolarWinds for users of its Orion network management software as far back as March 2020.

In its advisory on the VMware vulnerability, the NSA urged patching it “as soon as possible,” specifically encouraging the National Security System, Department of Defense, and defense contractors to make doing so a high priority.

The NSA said that in order to exploit this particular flaw, hackers would already need to have access to a vulnerable VMware device’s management interface — i.e., they would need to be on the target’s internal network (provided the vulnerable VMware interface was not accessible from the Internet). However, the SolarWinds compromise would have provided that internal access nicely.

In response to questions from KrebsOnSecurity, VMware said it has “received no notification or indication that the CVE 2020-4006 was used in conjunction with the SolarWinds supply chain compromise.”

VMware added that while some of its own networks used the vulnerable SolarWinds Orion software, an investigation has so far revealed no evidence of exploitation.

“While we have identified limited instances of the vulnerable SolarWinds Orion software in our environment, our own internal investigation has not revealed any indication of exploitation,” the company said in a statement. “This has also been confirmed by SolarWinds own investigations to date.”

On Dec. 17, DHS’s Cybersecurity and Infrastructure Security Agency (CISA) released a sobering alert on the SolarWinds attack, noting that CISA had evidence of additional access vectors other than the SolarWinds Orion platform.

CISA’s advisory specifically noted that “one of the principal ways the adversary is accomplishing this objective is by compromising the Security Assertion Markup Language (SAML) signing certificate using their escalated Active Directory privileges. Once this is accomplished, the adversary creates unauthorized but valid tokens and presents them to services that trust SAML tokens from the environment. These tokens can then be used to access resources in hosted environments, such as email, for data exfiltration via authorized application programming interfaces (APIs).”

Indeed, the NSA’s Dec. 7 advisory said the hacking activity it saw involving the VMware vulnerability “led to the installation of a web shell and follow-on malicious activity where credentials in the form of SAML authentication assertions were generated and sent to Microsoft Active Directory Federation Services (ADFS), which in turn granted the actors access to protected data.”

Also on Dec. 17, the NSA released a far more detailed advisory explaining how it has seen the VMware vulnerability being used to forge SAML tokens, this time specifically referencing the SolarWinds compromise.

Asked about the potential connection, the NSA said only that “if malicious cyber actors gain initial access to networks through the SolarWinds compromise, the TTPs [tactics, techniques and procedures] noted in our December 17 advisory may be used to forge credentials and maintain persistent access.”

“Our guidance in this advisory helps detect and mitigate against this, no matter the initial access method,” the NSA said.

CISA’s analysis suggested the crooks behind the SolarWinds intrusion were heavily focused on impersonating trusted personnel on targeted networks, and that they’d devised clever ways to bypass multi-factor authentication (MFA) systems protecting networks they targeted.

The bulletin references research released earlier this week by security firm Volexity, which described encountering the same attackers using a novel technique to bypass MFA protections provided by Duo for Microsoft Outlook Web App (OWA) users.

Duo’s parent Cisco Systems Inc. responded that the attack described by Volexity didn’t target any specific vulnerability in its products. As Ars Technica explained, the bypass involving Duo’s protections could have just as easily involved any of Duo’s competitors.

“MFA threat modeling generally doesn’t include a complete system compromise of an OWA server,” Ars’ Dan Goodin wrote. “The level of access the hacker achieved was enough to neuter just about any defense.”

Several media outlets, including The New York Times and The Washington Post, have cited anonymous government sources saying the group behind the SolarWinds hacks was known as APT29 or “Cozy Bear,” an advanced threat group believed to be part of the Russian Federal Security Service (FSB).

SolarWinds has said almost 18,000 customers may have received the backdoored Orion software updates. So far, only a handful of customers targeted by the suspected Russian hackers behind the SolarWinds compromise have been made public — including the U.S. Commerce, Energy and Treasury departments, and the DHS.

No doubt we will hear about new victims in the public and private sector in the coming days and weeks. In the meantime, thousands of organizations are facing incredibly costly, disruptive and time-intensive work in determining whether they were compromised and if so what to do about it.

The CISA advisory notes the attackers behind the SolarWinds compromises targeted key personnel at victim firms — including cyber incident response staff, and IT email accounts. The warning suggests organizations that suspect they were victims should assume their email communications and internal network traffic are compromised, and rely upon or build out-of-band systems for discussing internally how they will proceed to clean up the mess.

“If the adversary has compromised administrative level credentials in an environment—or if organizations identify SAML abuse in the environment, simply mitigating individual issues, systems, servers, or specific user accounts will likely not lead to the adversary’s removal from the network,” CISA warned. “In such cases, organizations should consider the entire identity trust store as compromised. In the event of a total identity compromise, a full reconstitution of identity and trust services is required to successfully remediate. In this reconstitution, it bears repeating that this threat actor is among the most capable, and in many cases, a full rebuild of the environment is the safest action.”

U.S. Treasury, Commerce Depts. Hacked Through SolarWinds Compromise

Communications at the U.S. Treasury and Commerce Departments were reportedly compromised by a supply chain attack on SolarWinds, a security vendor that helps the federal government and a range of Fortune 500 companies monitor the health of their IT networks. Given the breadth of the company’s customer base, experts say the incident may be just the first of many such disclosures.

Some of SolarWinds’ customers. Source: solarwinds.com

According to a Reuters story, hackers believed to be working for Russia have been monitoring internal email traffic at the U.S. Treasury and Commerce departments. Reuters reports the attackers were able to surreptitiously tamper with updates released by SolarWinds for its Orion platform, a suite of network management tools.

In a security advisory, Austin, Texas based SolarWinds acknowledged its systems “experienced a highly sophisticated, manual supply chain attack on SolarWinds Orion Platform software builds for versions 2019.4 HF 5 through 2020.2.1, released between March 2020 and June 2020.”

In response to the intrusions at Treasury and Commerce, the Department of Homeland Security’s Cybersecurity and Infrastructure Security Agency (CISA) took the unusual step of issuing an emergency directive ordering all federal agencies to immediately disconnect the affected Orion products from their networks.

“Treat all hosts monitored by the SolarWinds Orion monitoring software as compromised by threat actors and assume that further persistence mechanisms have been deployed,” CISA advised.

A blog post by Microsoft says the attackers were able to add malicious code to software updates provided by SolarWinds for Orion users. “This results in the attacker gaining a foothold in the network, which the attacker can use to gain elevated credentials,” Microsoft wrote.

From there, the attackers would be able to forge single sign-on tokens that impersonate any of the organization’s existing users and accounts, including highly privileged accounts on the network.

“Using highly privileged accounts acquired through the technique above or other means, attackers may add their own credentials to existing application service principals, enabling them to call APIs with the permission assigned to that application,” Microsoft explained.

Malicious code added to an Orion software update may have gone undetected by antivirus software and other security tools on host systems thanks in part to guidance from SolarWinds itself. In this support advisory, SolarWinds says its products may not work properly unless their file directories are exempted from antivirus scans and group policy object restrictions.

The Reuters story quotes several anonymous sources saying the intrusions at the Commerce and Treasury departments could be just the tip of the iceberg. That seems like a fair bet.

SolarWinds says it has over 300,000 customers including:

-more than 425 of the U.S. Fortune 500
-all ten of the top ten US telecommunications companies
-all five branches of the U.S. military
-all five of the top five U.S. accounting firms
-the Pentagon
-the State Department
-the National Security Agency
-the Department of Justice
-The White House.

It’s unclear how many of the customers listed on SolarWinds’ website are users of the affected Orion products. But Reuters reports the supply chain attack on SolarWinds is connected to a broad campaign that also involved the recently disclosed hack at FireEye, wherein hackers gained access to a slew of proprietary tools the company uses to help customers find security weaknesses in their computers and networks.

The compromises at the U.S. federal agencies are thought to date back to earlier this summer, and are being blamed on hackers working for the Russian government.

In its own advisory, FireEye said multiple updates poisoned with a malicious backdoor program were digitally signed with a SolarWinds certificate from March through May 2020, and posted to the SolarWindws update website.

FireEye posits the impact of the hack on SolarWinds is widespread, affecting public and private organizations around the world.

“The victims have included government, consulting, technology, telecom and extractive entities in North America, Europe, Asia and the Middle East,” the company’s analysts wrote. “We anticipate there are additional victims in other countries and verticals.”

Update, 8:30 p.m. ET: An earlier version of this story incorrectly stated that FireEye attributed the SolarWinds attack to APT29. That information has been removed from the story.

Ransomware Group Turns to Facebook Ads

It’s bad enough that many ransomware gangs now have blogs where they publish data stolen from companies that refuse to make an extortion payment. Now, one crime group has started using hacked Facebook accounts to run ads publicly pressuring their ransomware victims into paying up.

On the evening of Monday, Nov. 9, an ad campaign apparently taken out by the Ragnar Locker Team began appearing on Facebook. The ad was designed to turn the screws to the Italian beverage vendor Campari Group, which acknowledged on Nov. 3 that its computer systems had been sidelined by a malware attack.

On Nov. 6, Campari issued a follow-up statement saying “at this stage, we cannot completely exclude that some personal and business data has been taken.”

“This is ridiculous and looks like a big fat lie,” reads the Facebook ad campaign from the Ragnar crime group. “We can confirm that confidential data was stolen and we talking about huge volume of data.”

The ad went on to say Ragnar Locker Team had offloaded two terabytes of information and would give the Italian firm until 6 p.m. EST today (Nov. 10) to negotiate an extortion payment in exchange for a promise not to publish the stolen files.

The Facebook ad blitz was paid for by Hodson Event Entertainment, an account tied to Chris Hodson, a deejay based in Chicago. Contacted by KrebsOnSecurity, Hodson said his Facebook account indeed was hacked, and that the attackers had budgeted $500 for the entire campaign.

“I thought I had two-step verification turned on for all my accounts, but now it looks like the only one I didn’t have it set for was Facebook,” Hodson said.

Hodson said a review of his account shows the unauthorized campaign reached approximately 7,150 Facebook users, and generated 770 clicks, with a cost-per-result of 21 cents. Of course, it didn’t cost the ransomware group anything. Hodson said Facebook billed him $35 for the first part of the campaign, but apparently detected the ads as fraudulent sometime this morning before his account could be billed another $159 for the campaign.

The results of the unauthorized Facebook ad campaign. Image: Chris Hodson.

It’s not clear whether this was an isolated incident, or whether the fraudsters also ran ads using other hacked Facebook accounts. A spokesperson for Facebook said the company is still investigating the incident. A request for comment sent via email to Campari’s media relations team was returned as undeliverable.

But it seems likely we will continue to see more of this and other mainstream advertising efforts by ransomware groups going forward, even if victims really have no expectation that paying an extortion demand will result in criminals actually deleting or not otherwise using stolen data.

Fabian Wosar, chief technology officer at computer security firm Emsisoft, said some ransomware groups have become especially aggressive of late in pressuring their victims to pay up.

“They have also started to call victims,” Wosar said. “They’re outsourcing to Indian call centers, who call victims asking when they are going to pay or have their data leaked.”

The Now-Defunct Firms Behind 8chan, QAnon

Some of the world’s largest Internet firms have taken steps to crack down on disinformation spread by QAnon conspiracy theorists and the hate-filled anonymous message board 8chan. But according to a California-based security researcher, those seeking to de-platform these communities may have overlooked a simple legal solution to that end: Both the Nevada-based web hosting company owned by 8chan’s current figurehead and the California firm that provides its sole connection to the Internet are defunct businesses in the eyes of their respective state regulators.

In practical terms, what this means is that the legal contracts which granted these companies temporary control over large swaths of Internet address space are now null and void, and American Internet regulators would be well within their rights to cancel those contracts and reclaim the space.

The IP address ranges in the upper-left portion of this map of QAnon and 8kun-related sites — some 21,000 IP addresses beginning in “206.” and “207.” — are assigned to N.T. Technology Inc. Image source: twitter.com/Redrum_of_Crows

That idea was floated by Ron Guilmette, a longtime anti-spam crusader who recently turned his attention to disrupting the online presence of QAnon and 8chan (recently renamed “8kun”).

On Sunday, 8chan and a host of other sites related to QAnon conspiracy theories were briefly knocked offline after Guilmette called 8chan’s anti-DDoS provider and convinced them to stop protecting the site from crippling online attacks (8Chan is now protected by an anti-DDoS provider in St. Petersburg, Russia).

The public face of 8chan is Jim Watkins, a pig farmer in the Philippines who many experts believe is also the person behind the shadowy persona of “Q” at the center of the conspiracy theory movement.

Watkin owns and operates a Reno, Nev.-based hosting firm called N.T. Technology Inc. That company has a legal contract with the American Registry for Internet Numbers (ARIN), the non-profit which administers IP addresses for entities based in North America.

ARIN’s contract with N.T. Technology gives the latter the right to use more than 21,500 IP addresses. But as Guilmette discovered recently, N.T. Technology is listed in Nevada Secretary of State records as under an “administrative hold,” which according to Nevada statute is a “terminated” status indicator meaning the company no longer has the right to transact business in the state.

N.T. Technology’s listing in the Nevada Secretary of State records. Click to Enlarge.

The same is true for Centauri Communications, a Freemont, Calif.-based Internet Service Provider that serves as N.T. Technology’s colocation provider and sole connection to the larger Internet. Centauri was granted more than 4,000 IPv4 addresses by ARIN more than a decade ago.

According to the California Secretary of State, Centauri’s status as a business in the state is “suspended.” It appears that Centauri hasn’t filed any business records with the state since 2009, and the state subsequently suspended the company’s license to do business in Aug. 2012. Separately, the California State Franchise Tax Board (FTB) suspended this company as of April 1, 2014.

Centauri Communications’ listing with the California Secretary of State’s office.

Neither Centauri Communications nor N.T. Technology responded to repeated requests for comment.

KrebsOnSecurity shared Guilmette’s findings with ARIN, which said it would investigate the matter.

“ARIN has received a fraud report from you and is evaluating it,” a spokesperson for ARIN said. “We do not comment on such reports publicly.”

Guilmette said apart from reclaiming the Internet address space from Centauri and NT Technology, ARIN could simply remove each company’s listings from the global WHOIS routing records. Such a move, he said, would likely result in most ISPs blocking access to those IP addresses.

“If ARIN were to remove these records from the WHOIS database, it would serve to de-legitimize the use of these IP blocks by the parties involved,” he said. “And globally, it would make it more difficult for the parties to find people willing to route packets to and from those blocks of addresses.”

QAnon/8Chan Sites Briefly Knocked Offline

A phone call to an Internet provider in Oregon on Sunday evening was all it took to briefly sideline multiple websites related to 8chan/8kun — a controversial online image board linked to several mass shootings — and QAnon, the far-right conspiracy theory which holds that a cabal of Satanic pedophiles is running a global child sex-trafficking ring and plotting against President Donald Trump. Following a brief disruption, the sites have come back online with the help of an Internet company based in St. Petersburg, Russia.

The IP address range in the upper-right portion of this map of QAnon and 8kun-related sites — 203.28.246.0/24 — is assigned to VanwaTech and briefly went offline this evening. Source: twitter.com/Redrum_of_Crows.

A large number of 8kun and QAnon-related sites (see map above) are connected to the Web via a single Internet provider in Vancouver, Wash. called VanwaTech (a.k.a. “OrcaTech“). Previous appeals to VanwaTech to disconnect these sites have fallen on deaf ears, as the company’s owner Nick Lim reportedly has been working with 8kun’s administrators to keep the sites online in the name of protecting free speech.

But VanwaTech also had a single point of failure on its end: The swath of Internet addresses serving the various 8kun/QAnon sites were being protected from otherwise crippling and incessant distributed-denial-of-service (DDoS) attacks by Hillsboro, Ore. based CNServers LLC.

On Sunday evening, security researcher Ron Guilmette placed a phone call to CNServers’ owner, who professed to be shocked by revelations that his company was helping QAnon and 8kun keep the lights on.

Within minutes of that call, CNServers told its customer — Spartan Host Ltd., which is registered in Belfast, Northern Ireland — that it would no longer be providing DDoS protection for the set of 254 Internet addresses that Spartan Host was routing on behalf of VanwaTech.

Contacted by KrebsOnSecurity, the person who answered the phone at CNServers asked not to be named in this story for fear of possible reprisals from the 8kun/QAnon crowd. But they confirmed that CNServers had indeed terminated its service with Spartan Host. That person added they weren’t a fan of either 8kun or QAnon, and said they would not self-describe as a Trump supporter.

CNServers said that shortly after it withdrew its DDoS protection services, Spartan Host changed its settings so that VanwaTech’s Internet addresses were protected from attacks by ddos-guard[.]net, a company based in St. Petersburg, Russia.

Spartan Host’s founder, 25-year-old Ryan McCully, confirmed CNServers’ report. McCully declined to say for how long VanwaTech had been a customer, or whether Spartan Host had experienced any attacks as a result of CNServers’ action.

McCully said while he personally doesn’t subscribe to the beliefs espoused by QAnon or 8kun, he intends to keep VanwaTech as a customer going forward.

“We follow the ‘law of the land’ when deciding what we allow to be hosted with us, with some exceptions to things that may cause resource issues etc.,” McCully said in a conversation over instant message. “Just because we host something, it doesn’t say anything about we do and don’t support, our opinions don’t come into hosted content decisions.”

But according to Guilmette, Spartan Host’s relationship with VanwaTech wasn’t widely known previously because Spartan Host had set up what’s known as a “private peering” agreement with VanwaTech. That is to say, the two companies had a confidential business arrangement by which their mutual connections were not explicitly stated or obvious to other Internet providers on the global Internet.

Guilmette said private peering relationships often play a significant role in a good deal of behind-the-scenes-mischief when the parties involved do not want anyone else to know about their relationship.

“These arrangements are business agreements that are confidential between two parties, and no one knows about them, unless you start asking questions,” Guilmette said. “It certainly appears that a private peering arrangement was used in this instance in order to hide the direct involvement of Spartan Host in providing connectivity to VanwaTech and thus to 8kun. Perhaps Mr. McCully was not eager to have his involvement known.”

8chan, which rebranded last year as 8kun, has been linked to white supremacism, neo-Nazism, antisemitism, multiple mass shootings, and is known for hosting child pornography. After three mass shootings in 2019 revealed the perpetrators had spread their manifestos on 8chan and even streamed their killings live there, 8chan was ostracized by one Internet provider after another.

The FBI last year identified QAnon as a potential domestic terror threat, noting that some of its followers have been linked to violent incidents motivated by fringe beliefs.

Further reading:

What Is QAnon?

QAnon: A Timeline of Violent Linked to the Conspiracy Theory

Microsoft Uses Trademark Law to Disrupt Trickbot Botnet

Microsoft Corp. has executed a coordinated legal sneak attack in a bid to disrupt the malware-as-a-service botnet Trickbot, a global menace that has infected millions of computers and is used to spread ransomware. A court in Virginia granted Microsoft control over many Internet servers Trickbot uses to plunder infected systems, based on novel claims that the crime machine abused the software giant’s trademarks. However, it appears the operation has not completely disabled the botnet.

A spam email containing a Trickbot-infected attachment that was sent earlier this year. Image: Microsoft.

“We disrupted Trickbot through a court order we obtained as well as technical action we executed in partnership with telecommunications providers around the world,” wrote Tom Burt, corporate vice president of customer security and trust at Microsoft, in a blog post this morning about the legal maneuver. “We have now cut off key infrastructure so those operating Trickbot will no longer be able to initiate new infections or activate ransomware already dropped into computer systems.”

Microsoft’s action comes just days after the U.S. military’s Cyber Command carried out its own attack that sent all infected Trickbot systems a command telling them to disconnect themselves from the Internet servers the Trickbot overlords used to control them. The roughly 10-day operation by Cyber Command also stuffed millions of bogus records about new victims into the Trickbot database in a bid to confuse the botnet’s operators.

In legal filings, Microsoft argued that Trickbot irreparably harms the company “by damaging its reputation, brands, and customer goodwill. Defendants physically alter and corrupt Microsoft products such as the Microsoft Windows products. Once infected, altered and controlled by Trickbot, the Windows operating system ceases to operate normally and becomes tools for Defendants to conduct their theft.”

From the civil complaint Microsoft filed on October 6 with the U.S. District Court for the Eastern District of Virginia:

“However, they still bear the Microsoft and Windows trademarks. This is obviously meant to and does mislead Microsoft’s customers, and it causes extreme damage to Microsoft’s brands and trademarks.”

“Users subject to the negative effects of these malicious applications incorrectly believe that Microsoft and Windows are the source of their computing device problems. There is great risk that users may attribute this problem to Microsoft and associate these problems with Microsoft’s Windows products, thereby diluting and tarnishing the value of the Microsoft and Windows trademarks and brands.”

Microsoft said it will leverage the seized Trickbot servers to identify and assist Windows users impacted by the Trickbot malware in cleaning the malware off of their systems.

Trickbot has been used to steal passwords from millions of infected computers, and reportedly to hijack access to well more than 250 million email accounts from which new copies of the malware are sent to the victim’s contacts.

Trickbot’s malware-as-a-service feature has made it a reliable vehicle for deploying various strains of ransomware, locking up infected systems on a corporate network unless and until the company agrees to make an extortion payment.

A particularly destructive ransomware strain that is closely associated with Trickbot — known as “Ryuk” or “Conti” — has been responsible for costly attacks on countless organizations over the past year, including healthcare providers, medical research centers and hospitals.

One recent Ryuk victim is Universal Health Services (UHS), a Fortune 500 hospital and healthcare services provider that operates more than 400 facilities in the U.S. and U.K.

On Sunday, Sept. 27, UHS shut down its computer systems at healthcare facilities across the United States in a bid to stop the spread of the malware. The disruption caused some of the affected hospitals to redirect ambulances and relocate patients in need of surgery to other nearby hospitals.

Microsoft said it did not expect its action to permanently disrupt Trickbot, noting that the crooks behind the botnet will likely make efforts to revive their operations. But so far it’s not clear whether Microsoft succeeded in commandeering all of Trickbot’s control servers, or when exactly the coordinated seizure of those servers occurred.

As the company noted in its legal filings, the set of Internet address used as Trickbot controllers is dynamic, making attempts to disable the botnet more challenging.

Indeed, according to real-time information posted by Feodo Tracker, a Swiss security site that tracks Internet servers used as controllers for Trickbot and other botnets, nearly two dozen Trickbot control servers — some of which first went active at beginning of this month — are still live and responding to requests at the time of this publication.

Trickbot control servers that are currently online. Source: Feodotracker.abuse.ch

Sendgrid Under Siege from Hacked Accounts

Email service provider Sendgrid is grappling with an unusually large number of customer accounts whose passwords have been cracked, sold to spammers, and abused for sending phishing and email malware attacks. Sendgrid’s parent company Twilio says it is working on a plan to require multi-factor authentication for all of its customers, but that solution may not come fast enough for organizations having trouble dealing with the fallout in the meantime.

Image: Wikipedia

Many companies use Sendgrid to communicate with their customers via email, or else pay marketing firms to do that on their behalf using Sendgrid’s systems. Sendgrid takes steps to validate that new customers are legitimate businesses, and that emails sent through its platform carry the proper digital signatures that other companies can use to validate that the messages have been authorized by its customers.

But this also means when a Sendgrid customer account gets hacked and used to send malware or phishing scams, the threat is particularly acute because a large number of organizations allow email from Sendgrid’s systems to sail through their spam-filtering systems.

To make matters worse, links included in emails sent through Sendgrid are obfuscated (mainly for tracking deliverability and other metrics), so it is not immediately clear to recipients where on the Internet they will be taken when they click.

Dealing with compromised customer accounts is a constant challenge for any organization doing business online today, and certainly Sendgrid is not the only email marketing platform dealing with this problem. But according to multiple emails from readers, recent threads on several anti-spam discussion lists, and interviews with people in the anti-spam community, over the past few months there has been a marked increase in malicious, phishous and outright spammy email being blasted out via Sendgrid’s servers.

Rob McEwen is CEO of Invaluement.com, an anti-spam firm whose data on junk email trends are used to improve the spam-blocking technologies deployed by several Fortune 100 companies. McEwen said no other email service provider has come close to generating the volume of spam that’s been emanating from Sendgrid accounts lately.

“As far as the nasty criminal phishes and viruses, I think there’s not even a close second in terms of how bad it’s been with Sendgrid over the past few months,” he said.

Trying to filter out bad emails coming from a major email provider that so many legitimate companies rely upon to reach their customers can be a dicey business. If you filter the emails too aggressively you end up with an unacceptable number of “false positives,” i.e., benign or even desirable emails that get flagged as spam and sent to the junk folder or blocked altogether.

But McEwen said the incidence of malicious spam coming from Sendgrid has gotten so bad that he recently launched a new anti-spam block list specifically to filter out email from Sendgrid accounts that have been known to be blasting large volumes of junk or malicious email.

“Before I implemented this in my own filtering system a week ago, I was getting three to four phone calls or stern emails a week from angry customers wondering why these malicious emails were getting through to their inboxes,” McEwen said. “And I just am not seeing anything this egregious in terms of viruses and spams from the other email service providers.”

In an interview with KrebsOnSecurity, Sendgrid parent firm Twilio acknowledged the company had recently seen an increase in compromised customer accounts being abused for spam. While Sendgrid does allow customers to use multi-factor authentication (also known as two-factor authentication or 2FA), this protection is not mandatory.

But Twilio Chief Security Officer Steve Pugh said the company is working on changes that would require customers to use some form of 2FA in addition to usernames and passwords.

“Twilio believes that requiring 2FA for customer accounts is the right thing to do, and we’re working towards that end,” Pugh said. “2FA has proven to be a powerful tool in securing communications channels. This is part of the reason we acquired Authy and created a line of account security products and services. Twilio, like other platforms, is forming a plan on how to better secure our customers’ accounts through native technologies such as Authy and additional account level controls to mitigate known attack vectors.”

Requiring customers to use some form of 2FA would go a long way toward neutralizing the underground market for compromised Sendgrid accounts, which are sold by a variety of cybercriminals who specialize in gaining access to accounts by targeting users who re-use the same passwords across multiple websites.

One such individual, who goes by the handle “Kromatix” on several forums, is currently selling access to more than 400 compromised Sendgrid user accounts. The pricing attached to each account is based on volume of email it can send in a given month. Accounts that can send up to 40,000 emails a month go for $15, whereas those capable of blasting 10 million missives a month sell for $400.

“I have a large supply of cracked Sendgrid accounts that can be used to generate an API key which you can then plug into your mailer of choice and send massive amounts of emails with ensured delivery,” Kromatix wrote in an Aug. 23 sales thread. “Sendgrid servers maintain a very good reputation with [email service providers] so your content becomes much more likely to get into the inbox so long as your setup is correct.”

Neil Schwartzman, executive director of the anti-spam group CAUCE, said Sendgrid’s 2FA plans are long overdue, noting that the company bought Authy back in 2015.

Single-factor authentication for a company like this in 2020 is just ludicrous given the potential damage and malicious content we’re seeing,” Schwartzman said.

“I understand that it’s a task to invoke 2FA, and given the volume of customers Sendgrid has that’s something to consider because there’s going to be a lot of customer overhead involved,” he continued. “But it’s not like your bank, social media account, email and plenty of other places online don’t already insist on it.”

Schwartzman said if Twilio doesn’t act quickly enough to fix the problem on its end, the major email providers of the world (think Google, Microsoft and Apple) — and their various machine-learning anti-spam algorithms — may do it for them.

“There is a tipping point after which receiving firms start to lose patience and start to more aggressively filter this stuff,” he said. “If seeing a Sendgrid email according to machine learning becomes a sign of abuse, trust me the machines will make the decisions even if the people don’t.”

Voice Phishers Targeting Corporate VPNs

The COVID-19 epidemic has brought a wave of email phishing attacks that try to trick work-at-home employees into giving away credentials needed to remotely access their employers’ networks. But one increasingly brazen group of crooks is taking your standard phishing attack to the next level, marketing a voice phishing service that uses a combination of one-on-one phone calls and custom phishing sites to steal VPN credentials from employees.

According to interviews with several sources, this hybrid phishing gang has a remarkably high success rate, and operates primarily through paid requests or “bounties,” where customers seeking access to specific companies or accounts can hire them to target employees working remotely at home.

And over the past six months, the criminals responsible have created dozens if not hundreds of phishing pages targeting some of the world’s biggest corporations. For now at least, they appear to be focusing primarily on companies in the financial, telecommunications and social media industries.

“For a number of reasons, this kind of attack is really effective,” said Allison Nixon, chief research officer at New York-based cyber investigations firm Unit 221B. “Because of the Coronavirus, we have all these major corporations that previously had entire warehouses full of people who are now working remotely. As a result the attack surface has just exploded.”

TARGET: NEW HIRES

A typical engagement begins with a series of phone calls to employees working remotely at a targeted organization. The phishers will explain that they’re calling from the employer’s IT department to help troubleshoot issues with the company’s virtual private networking (VPN) technology.

The employee phishing page bofaticket[.]com. Image: urlscan.io

The goal is to convince the target either to divulge their credentials over the phone or to input them manually at a website set up by the attackers that mimics the organization’s corporate email or VPN portal.

Zack Allen is director of threat intelligence for ZeroFOX, a Baltimore-based company that helps customers detect and respond to risks found on social media and other digital channels. Allen has been working with Nixon and several dozen other researchers from various security firms to monitor the activities of this prolific phishing gang in a bid to disrupt their operations.

Allen said the attackers tend to focus on phishing new hires at targeted companies, and will often pose as new employees themselves working in the company’s IT division. To make that claim more believable, the phishers will create LinkedIn profiles and seek to connect those profiles with other employees from that same organization to support the illusion that the phony profile actually belongs to someone inside the targeted firm.

“They’ll say ‘Hey, I’m new to the company, but you can check me out on LinkedIn’ or Microsoft Teams or Slack, or whatever platform the company uses for internal communications,” Allen said. “There tends to be a lot of pretext in these conversations around the communications and work-from-home applications that companies are using. But eventually, they tell the employee they have to fix their VPN and can they please log into this website.”

SPEAR VISHING

The domains used for these pages often invoke the company’s name, followed or preceded by hyphenated terms such as “vpn,” “ticket,” “employee,” or “portal.” The phishing sites also may include working links to the organization’s other internal online resources to make the scheme seem more believable if a target starts hovering over links on the page.

Allen said a typical voice phishing or “vishing” attack by this group involves at least two perpetrators: One who is social engineering the target over the phone, and another co-conspirator who takes any credentials entered at the phishing page and quickly uses them to log in to the target company’s VPN platform in real-time.

Time is of the essence in these attacks because many companies that rely on VPNs for remote employee access also require employees to supply some type of multi-factor authentication in addition to a username and password — such as a one-time numeric code generated by a mobile app or text message. And in many cases, those codes are only good for a short duration — often measured in seconds or minutes.

But these vishers can easily sidestep that layer of protection, because their phishing pages simply request the one-time code as well.

A phishing page (helpdesk-att[.]com) targeting AT&T employees. Image: urlscan.io

Allen said it matters little to the attackers if the first few social engineering attempts fail. Most targeted employees are working from home or can be reached on a mobile device. If at first the attackers don’t succeed, they simply try again with a different employee.

And with each passing attempt, the phishers can glean important details from employees about the target’s operations, such as company-specific lingo used to describe its various online assets, or its corporate hierarchy.

Thus, each unsuccessful attempt actually teaches the fraudsters how to refine their social engineering approach with the next mark within the targeted organization, Nixon said.

“These guys are calling companies over and over, trying to learn how the corporation works from the inside,” she said.

NOW YOU SEE IT, NOW YOU DON’T

All of the security researchers interviewed for this story said the phishing gang is pseudonymously registering their domains at just a handful of domain registrars that accept bitcoin, and that the crooks typically create just one domain per registrar account.

“They’ll do this because that way if one domain gets burned or taken down, they won’t lose the rest of their domains,” Allen said.

More importantly, the attackers are careful to do nothing with the phishing domain until they are ready to initiate a vishing call to a potential victim. And when the attack or call is complete, they disable the website tied to the domain.

This is key because many domain registrars will only respond to external requests to take down a phishing website if the site is live at the time of the abuse complaint. This requirement can stymie efforts by companies like ZeroFOX that focus on identifying newly-registered phishing domains before they can be used for fraud.

“They’ll only boot up the website and have it respond at the time of the attack,” Allen said. “And it’s super frustrating because if you file an abuse ticket with the registrar and say, ‘Please take this domain away because we’re 100 percent confident this site is going to be used for badness,’ they won’t do that if they don’t see an active attack going on. They’ll respond that according to their policies, the domain has to be a live phishing site for them to take it down. And these bad actors know that, and they’re exploiting that policy very effectively.”

A phishing page (github-ticket[.]com) aimed at siphoning credentials for a target organization’s access to the software development platform Github. Image: urlscan.io

SCHOOL OF HACKS

Both Nixon and Allen said the object of these phishing attacks seems to be to gain access to as many internal company tools as possible, and to use those tools to seize control over digital assets that can quickly be turned into cash. Primarily, that includes any social media and email accounts, as well as associated financial instruments such as bank accounts and any cryptocurrencies.

Nixon said she and others in her research group believe the people behind these sophisticated vishing campaigns hail from a community of young men who have spent years learning how to social engineer employees at mobile phone companies and social media firms into giving up access to internal company tools.

Traditionally, the goal of these attacks has been gaining control over highly-prized social media accounts, which can sometimes fetch thousands of dollars when resold in the cybercrime underground. But this activity gradually has evolved toward more direct and aggressive monetization of such access.

On July 15, a number of high-profile Twitter accounts were used to tweet out a bitcoin scam that earned more than $100,000 in a few hours. According to Twitter, that attack succeeded because the perpetrators were able to social engineer several Twitter employees over the phone into giving away access to internal Twitter tools.

Nixon said it’s not clear whether any of the people involved in the Twitter compromise are associated with this vishing gang, but she noted that the group showed no signs of slacking off after federal authorities charged several people with taking part in the Twitter hack.

“A lot of people just shut their brains off when they hear the latest big hack wasn’t done by hackers in North Korea or Russia but instead some teenagers in the United States,” Nixon said. “When people hear it’s just teenagers involved, they tend to discount it. But the kinds of people responsible for these voice phishing attacks have now been doing this for several years. And unfortunately, they’ve gotten pretty advanced, and their operational security is much better now.”

A phishing page (vzw-employee[.]com) targeting employees of Verizon. Image: DomainTools

PROPER ADULT MONEY-LAUNDERING

While it may seem amateurish or myopic for attackers who gain access to a Fortune 100 company’s internal systems to focus mainly on stealing bitcoin and social media accounts, that access — once established — can be re-used and re-sold to others in a variety of ways.

“These guys do intrusion work for hire, and will accept money for any purpose,” Nixon said. “This stuff can very quickly branch out to other purposes for hacking.”

For example, Allen said he suspects that once inside of a target company’s VPN, the attackers may try to add a new mobile device or phone number to the phished employee’s account as a way to generate additional one-time codes for future access by the phishers themselves or anyone else willing to pay for that access.

Nixon and Allen said the activities of this vishing gang have drawn the attention of U.S. federal authorities, who are growing concerned over indications that those responsible are starting to expand their operations to include criminal organizations overseas.

“What we see now is this group is really good on the intrusion part, and really weak on the cashout part,” Nixon said. “But they are learning how to maximize the gains from their activities. That’s going to require interactions with foreign gangs and learning how to do proper adult money laundering, and we’re already seeing signs that they’re growing up very quickly now.”

WHAT CAN COMPANIES DO?

Many companies now make security awareness and training an integral part of their operations. Some firms even periodically send test phishing messages to their employees to gauge their awareness levels, and then require employees who miss the mark to undergo additional training.

Such precautions, while important and potentially helpful, may do little to combat these phone-based phishing attacks that tend to target new employees. Both Allen and Nixon — as well as others interviewed for this story who asked not to be named — said the weakest link in most corporate VPN security setups these days is the method relied upon for multi-factor authentication.

A U2F device made by Yubikey, plugged into the USB port on a computer.

One multi-factor option — physical security keys — appears to be immune to these sophisticated scams. The most commonly used security keys are inexpensive USB-based devices. A security key implements a form of multi-factor authentication known as Universal 2nd Factor (U2F), which allows the user to complete the login process simply by inserting the USB device and pressing a button on the device. The key works without the need for any special software drivers.

The allure of U2F devices for multi-factor authentication is that even if an employee who has enrolled a security key for authentication tries to log in at an impostor site, the company’s systems simply refuse to request the security key if the user isn’t on their employer’s legitimate website, and the login attempt fails. Thus, the second factor cannot be phished, either over the phone or Internet.

In July 2018, Google disclosed that it had not had any of its 85,000+ employees successfully phished on their work-related accounts since early 2017, when it began requiring all employees to use physical security keys in place of one-time codes.

Probably the most popular maker of security keys is Yubico, which sells a basic U2F Yubikey for $20. It offers regular USB versions as well as those made for devices that require USB-C connections, such as Apple’s newer Mac OS systems. Yubico also sells more expensive keys designed to work with mobile devices. [Full disclosure: Yubico was recently an advertiser on this site].

Nixon said many companies will likely balk at the price tag associated with equipping each employee with a physical security key. But she said as long as most employees continue to work remotely, this is probably a wise investment given the scale and aggressiveness of these voice phishing campaigns.

“The truth is some companies are in a lot of pain right now, and they’re having to put out fires while attackers are setting new fires,” she said. “Fixing this problem is not going to be simple, easy or cheap. And there are risks involved if you somehow screw up a bunch of employees accessing the VPN. But apparently these threat actors really hate Yubikey right now.”