Investigators Find Encryption, Monitoring, Logging and Whitelisting Failures
Inadequate monitoring of databases and privileged accounts, incomplete multi-factor authentication and insufficient use of encryption: These are among the catalog of errors cited by British privacy regulators investigating the failures that contributed to the massive data breach involving Marriott’s Starwood guest reservation system.
Maryland-based global hotel chain Marriott suffered one of the worst data breaches in history, which only came to light only in 2018, four years after it began. By then, an estimated 339 million individuals’ personal details had been exposed.
A lengthy investigation by Britain’s Information Commissioner’s Office – on behalf of all EU privacy authorities – identified multiple deficiencies that contributed to the breach of the Starwood system that began in 2014, continued even after Marriott acquired Starwood in 2016, and wasn’t discovered until 2018 (see: Marriott Breach Takeaway: The M&A Cybersecurity Challenge).
“The ICO’s investigation found that there were failures by Marriott to put appropriate technical or organizational measures in place to protect the personal data being processed on its systems, as required by the General Data Protection Regulation,” the ICO says in its 91-page final penalty notice. Marriott also failed to notify the ICO within 72 hours of detecting the breach.
The ICO’s penalty notice cites “four principal failures” that appeared to have been exploited by more than one group of attackers:
- Insufficient monitoring of privileged accounts;
- Insufficient monitoring of databases;
- Poor controls for critical systems;
- Insufficient encryption.
Here are nine takeaways from the ICO’s probe and failures it identified.
1. Attacker Installed Web Shell
An attacker hit Starwood’s infrastructure on July 29, 2014, installing a web shell on a system tied to its Accolade software application, which “was used by Starwood to allow employees to request changes to any content on Starwood’s website,” the ICO’s final penalty notice states.
Using the web shell, the attacker gained direct access to a server and installed a remote access Trojan to maintain persistent access. Later “on an undetermined date,” the attacker deployed open source Mimikatz software to steal passwords, and memory-scraping malware to steal payment card details, investigators say. Using the stolen passwords, attackers were able “to continue to compromise user accounts,” conduct further reconnaissance and ultimately to dump data.
2. Database Dumps Continued
Starting in April 2015, or about nine months after installing the web shell, investigators say attackers created a number of files – such as “Reservation_Room_sharer.dmp” and “Consumption_Roomtype.dmp” – potentially “with a view to exfiltrating all the data contained in this table at once.”
Around Dec. 31, 2016 – after Marriott had acquired Starwood – attackers installed memory-scraping malware, which searches devices for payment-card data, on Starwood systems. After executing the malware on Jan. 10, 2017, attackers failed to steal any payment card data, the ICO says. “Marriott believes, but cannot be certain, that this action was carried out by a different attacker to the one responsible” for the prior data dumping, the ICO notes.
3. Database Count Finally Triggered Alarm
How did Marriott ultimately spot the long-running intrusion in 2018? Here’s a timeline of the events surrounding it being detected, as detailed by the ICO:
- Sept. 7, 2018: An attacker “performed a ‘count’ on the ‘Guest_Master_profile’ table, which would have told the attacker how many rows the table contained.” This count triggered an alert in its IBM Guardium database security software. The ICO notes that prior malicious activity involving the same database had not been spotted, because Marriott only configured Guardium to monitor fields containing payment card data.
- Sept. 9: Accenture, which was managing the Starwood Guest Reservation Database, contacted Marriott’s team to relay what was the first-ever Guardium alert the hotel chain received that was tied the breach.
- Sept. 10: Attackers appeared to still be at work, and exported a “PP_Master” table onto the Starwood system, apparently to exfiltrate the data.
- Sept. 9 or 10: The ICO says Marriott launched its “information security and privacy incident response” plan and began deploying real-time monitoring and forensic tools onto 70,000 legacy Starwood devices. “The purpose of this measure was to monitor the local system and identify potentially malicious activity in real-time, with findings reported back to Marriott’s central monitoring server,” the ICO says.
- Around Sept. 15: Marriott’s investigators identified unauthorized activity dating from two months before – on July 7 – and involving legitimate credentials stolen from Accenture employees.
- Sept. 17: Investigators identified a RAT in their systems, which they mitigated by blocking the IP address it was using to receive command-and-control instructions.
- Sometime in October: Investigators found that attackers had used Mimikatz multiple times, and also identified that payment-card-scraping malware had been deployed.
- Oct. 29: Hotel chain alerts the FBI to the attack.
- Nov. 13: Marriott begins finding multiple compressed, encrypted and previously deleted files created by attackers that contained massive exports of customer data.
- Nov. 22: Marriott notifies the ICO about the breach. Eight days later, it provides a follow-up report.
- Nov. 30: Marriott begins emailing notifications to victims, including a link to a dedicated website with details about how to contact a dedicated call center.
4. Encrypted and Unencrypted Data Stolen
Marriott estimates that 339 million customers were affected, with many seeing their unencrypted name, address, details of their visits – including the number of children they were traveling with – as well as email addresses and flight numbers, the ICO notes.
About 9.1 million encrypted payment cards were also exposed, as were 18.5 million passport numbers. Marriott said it was unlikely that all could be decrypted, as attackers did not appear to have obtained the decryption key. But 5.25 million customers’ passport numbers were stored in unencrypted format and were exposed.
5. Gaps in Cardholder Data Environment
The ICO’s probe identified multiple gaps in Marriott’s cardholder data environment, or CDE, including a failure to use multi-factor authentication to protect all accounts and systems with access to the CDE. “Marriott has explained that it believed that MFA was in place across the CDE because it had received assurances from Starwood’s management to this effect,” the ICO’s report notes.
Excerpt from the ICO’s final penalty notice to Marriott on Oct. 30, 2020, which references its initial “notice of intent” – NOI – to fine Marriott, issued on Aug. 23, 2019.
“This belief was corroborated by two reports on compliance issued independent PCI DSS assessor,” both pre-acquisition and post-acquisition, it adds. The Payment Card Industry Data Security Standard is meant to ensure that organizations that handle payment card data maintain a security environment.
The ICO says Marriott’s reliance on the findings of PCI DSS assessors – including a post-acquisition assessment dated May 23, 2017 – did not breach its GDPR obligations. Accordingly, it says that none of the penalty it imposed on Marriott pertained to the incomplete use of MFA.
Marriott did not immediately respond to a request to identify its PCI DSS assessors.
6. Privileged Accounts: Insufficient Monitoring
The first of the four failures identified by the ICO in its penalty notice was insufficient monitoring of privileged accounts.
Setting aside the incomplete MFA protection for the CDE, the ICO says that once attackers did gain access, “appropriate and adequate measures were not in place to allow for the identification of the breach and to further prevent unauthorized activity.” Specifically, it says Marriott should have been monitoring user activity, especially for privileged accounts, and taking a layered approach to security.
An incident report prepared by Verizon, dated April 11, 2019, notes that Marriott had not configured logging for systems and applications inside the CDE, which the ICO says would have been a core component of its monitoring regimen, to help identity suspicious activity, including via legitimate accounts suborned by attackers. The ICO says this failure breached Marriott’s GDPR obligations.
7. Databases: Insufficient Monitoring
In addition, “Marriott failed to adequately monitor the databases within the CDE,” the ICO says, citing deficient alerts, failing to aggregate logs, and failing “to log actions taken on the CDE system, such as the creation of files and the exporting of entire database tables.”
While Marriott employed some logging and fed this information to its security incident event management – aka SIEM – system, the ICO found failures in the database alerts it had set, as well as a failure to aggregate logs and “to log actions taken on the CDE system, such as the creation of files and exporting of entire database tables.”
“That Marriott did not detect the attack until alerted by Guardium is indicative of Marriott failing regularly to test, assess and evaluate the effectiveness of its security measures.”
— Information Commissioner’s Office
The ICO notes that Marriott’s IBM Guardium software could have been set to log all CDE database activities, including whenever a new file was created – for example, by hackers dumping database tables to new files. But Marriott was not doing such logging, which “rendered the SIEM and SOC” – security operations center – “ineffective.” In addition, it says Marriott failed to sufficiently log other operations, including “firewall and access logs.”
While such logging would not have prevented the attack, the ICO notes that it would have allowed Marriott to detect it much more quickly. “That Marriott did not detect the attack until alerted by Guardium is indicative of Marriott failing regularly to test, assess and evaluate the effectiveness of its security measures.”
The ICO also criticized Marriott’s all-or-nothing approach to monitoring – namely, only setting Guardium to monitor tables containing payment card data. Had Marriott taken a true risk-based approach, the ICO suggests that it would have set alerts to protect other sensitive information, including passport numbers. Instead, the organization appears to have been relying solely on MFA to ensure no unauthorized individuals accessed the CDE – an approach that failed.
8. Critical System Controls: Deficient
Beyond better “monitoring and security alerts, it would have been appropriate for Marriott to implement a form of server hardening as a preventive measure, which could have prevented the attacker from gaining access to administrator accounts and performing reconnaissance before traversing across a network,” the ICO says.
Citing guidance from the U.K.’s National Cyber Security Centre, the ICO notes that whitelisting – aka allow/deny lists – could have been set to restrict access to critical resources, based on user ID and IP address. (see: Forget Whitelists and Blacklists: Go for ‘Allow’ or ‘Deny’).
“In this incident, whitelisting could have aided in halting the reconnaissance and privilege escalation stage of the attack,” the ICO says. Likewise, whitelisting could have been used to prevent memory-scraping malware from being installed on point-of-sale terminals, it adds. The ICO also said that eliminating “outdated/obsolete software” from Marriott’s infrastructure – as recommended by PCI DSS auditors in 2017 and 2018 – would have helped to harden the environment against hack attacks.
9. Encryption: Incorrectly Configured
The Starwood reservation system included Oracle databases, which enabled Marriott to encrypt data using AES-128, “and it was Marriott’s responsibility to ensure this was configured correctly,” the ICO says.
But while the system was set up to encrypt payment card data, it was not encrypting other sensitive information, including some passport numbers, the ICO says.
In a heavily redacted section of the penalty notice, the ICO also notes that “the level of security that the encryption could have achieved was compromised within the Starwood guest database by a script, developed by Starwood which allowed for AES-128 encrypted entries in a database table to be decrypted.”
A footnote to the redacted section states: “While the [ICO] agrees that it is unlikely that the attacker did run the script millions of times, if the attacker so wished, this could have been achieved in very little time as it could run as an automated process.”
Whether attackers utilized this script is unclear.
But per its remit under GDPR and ability to fine organizations that failed to sufficiently safeguard Europeans’ personal information, the ICO’s report does not hesitate to call out the many different security, technical and organizational deficiencies investigators found at Marriott, even if every single one was not exploited by attackers.