Expired Credentials, Server Outage Lead to Inaccurate Tally
An expired digital certificate for Quest Diagnostics, a major test provider, and several technology woes temporarily prevented the state of California from receiving timely COVID-19 lab test data, resulting in an inaccurate tally of cases.
The expired certificate for Quest, plus a July 25 server outage and other technical issues – caused an inaccurate accounting of the number of COVID-19 cases in the state for several days in late July and early August, according to the Los Angeles Times.
The credential mishap prevented Quest from sending test results to the state from July 31 through Aug. 4 when the state also was experiencing other technical problems with its California Reportable Disease Information Exchange.
CalREDIE is a secure system for electronic disease reporting and surveillance. The system – which is used by about 60 local health departments and 350 laboratories – is meant to support 24×7 reporting and receipt of notifiable conditions, including COVID-19.
“The CalREDIE system was not built for the volume of data we are receiving as a result of the pandemic, which has resulted in labor-intensive process, and sometimes delays, for identifying positive COVID-19 cases,” the California Department of Public Health said in a statement provided to Information Security Media Group.
Plus, the state neglected to renew the Quest Diagnostics certificate in a timely manner, the statement notes.
“In the short term, we have reversed the technical changes that created the recent backlog of cases and renewed the certificate to ensure all laboratory data is being received,” the health department says.
In addition, the department says it has made upgrades to its servers to ensure they have extra capacity as well as a redundant system for validating the data and reports.
“We strengthened oversight and monitoring, so case data are complete, timely and validated. Since those changes were made, the system has been working as expected,” the state health department tells ISMG.
The state also is developing a new laboratory reporting system for COVID-19 to securely and accurately collect, store, analyze and publish electronic lab reporting and case data, the health department says.
“We have issued a procurement requesting innovative technology solutions for this system and will choose a vendor in early September. Once a vendor is chosen, we will have an estimated cost and timeline,” the statement says.
In the meantime, the state is updating its data and COVID-19 test positivity rates and is continuing to process the backlogged data, the department says.
Quest Diagnostics Reacts
Meanwhile, Quest Diagnostics says in a statement provided to ISMG that the delay in COVID-19 test result transmission was caused by the outdated certificate.
“For the California Department of Public Health to receive public health results, CalREDIE’s Public Health Information Networking System – PHINMS – certificate must be valid. This is to ensure public health message security,” Quest says. The lab’s certificate lapsed in late July, stopping test result transmissions until it was renewed, Quest adds.
“During this time, Quest was sending results, but American Public Health Laboratory Information System was unable to authenticate their receiver, which in this case was CalREDIE,” Quest says. This created the delay in result transmission, it adds.
For instance, one of the largest data breaches – the 2017 cyberattack on credit bureau Equifax, which exposed data pertaining to 148 million individuals in the U.S., 15 million in the U.K. and 20,000 in Canada – also involved expired credentials.
A 2018 report from the U.S. House of Representative’s Committee on Oversight and Government Reform notes that Equifax failed to catch the large exfiltration of data by hackers for months because a security certificate on a traffic monitoring device had expired. The breach was immediately detected on July 29, 2017, when Equifax updated the security certificate, the report notes.
“Certificates provide trust – a system/entity uses it to verify that the endpoint it is talking to is in fact the correct system/entity,” notes Jeremy Molnar, a senior vice president at security and privacy consulting firm CynergisTek.
“An expired certificate means that [the certificate’s] state has not been recently validated and the endpoint may no longer be the correct one. For many systems that require a validated certificate status, this means that process/transmission is stopped – i.e., do not send data to an untrusted endpoint – which is basically what happened in California,” he says.
A February report from research firm AIR Worldwide estimated that data losses from machine identity protection security breaches cost the global economy $51 billion to $72 billion annually.
“Like humans, every kind of machine – clouds, devices, web sites, applications, microservices – needs an identity authenticated and communicated privately,” notes Kevin Bocek, vice president of security strategy and threat intelligence at security vendor Venafi.
“One type of machine identity is a digital certificate. When a digital certificate expires, that machine can’t be trusted and all communication stops. This is why the certificates that serve as machine identities are the foundation of security and critical to keeping our digital world working,” he says.
Certificates all have a built-in expiration date that is between 90 days and two years, Bocek says. “Driven by a change from Apple, the maximum certificate lifespan will be changing in September to just 13 months – this is going to result in more outages like the one that affected the state of California. Certificates have to be replaced before they expire or that machine or application will stop communicating.”
The number of devices in organizations that need machine identities is skyrocketing, Bocek says. “They are used on all types of devices, including IoT and smart devices; they’re also used in applications, cloud workloads, containers and many algorithms.”
Molnar notes that organizations can choose to trust an expired certificate – “maybe assuming that the expiration date is the only problem with the certificate.”
He cautions, however, that this can “open you up to all sorts of attacks, such as a man-in-the-middle attacks, simply because an attacker is able to take advantage of bypassing the trust model. An expired certificate is basically the same as having no certificate in this instance. The big exception to that is that there may be an assumption of security simply because the certificate is used regardless of its validity.”
Certificates expire based on how long a particular certificate authority is willing to sign-off on the authenticity of a particular asset, he notes.
“Those systems at the top level of the public key infrastructure that provide the certificates often have longer valid dates than those at the bottom such as endpoints,” Molnar says. “A lot of this has to do with the amount of security expected to be in place to protect the CAs vs. how hard it can be to protect the endpoints and to ensure that validation provided by the certificate has meaning.”
Problems with expired certificates often occur because of a lack of a formal certificate management process, Molnar says.
Once an organization starts using certificates, it needs to be able to track when they expire and spell out how to request new/updated certificates, he notes.
“Often, organizations don’t often know they have a problem until the certificate actually expires,” he says. “At that point, they then have to scramble to get a new valid certificate and to then get it installed in the appropriate locations.”
Preventing Expired Certificate Issues
So, how can entities avoid mishaps involving expired certificates?
“We need the security community to pay more attention to machine identities to prevent these kinds of outages,” Bocek says. “For most organizations, the first thing they need to do is get visibility of all the certificates on their networks and clouds and when they will expire. Ultimately, automation is the answer to stop outages and make digital transformation successful.”
Molnar says organizations should establish a program for managing the certificates that are in use. “As a requestor, this can be anything from a basic spreadsheet to the many potential solutions available. If an organization decides to implement its own PKI, then it is significantly more complicated, but it still comes down to having a strong certificate management program.”
A formalized certificate management program should address expiring certificates before they become a problem, Molnar says.
“That is assuming you know about the certificates in the first place. So, having asset management as it relates to certificates is also important so you know which applications/systems are using certificates.”
Organizations also should conduct a business impact analysis or a related criticality analysis, he says.
“This allows an organization to know which are the most critical assets/processes and what they rely on to function/communicate. This should identify which processes are reliant on certificates so that the appropriate prioritization can be applied to managing those more critical practices and minimizing any potential negative impacts.”
Another good practice is to use multiple CAs, Molnar suggests. “The reason for this is that if a vendor were to have a breach or issue that breaks the trust provided by the CA, it will minimize the potential impacts by only invalidating a select number of certificates versus all of them.”