As America struggles to assess the damage from the
devastating SolarWinds cyberattack discovered in December, ProPublica has
learned of a promising defense that could shore up the vulnerability the
hackers exploited: a system the federal government funded but has never
required its vendors to use.
The massive breach, which U.S. intelligence agencies say was
“likely Russian in origin,” penetrated the computer systems of critical federal
agencies, including the Department of Homeland Security, the Treasury Department,
the National Institutes of Health and the Department of Justice, as well as a
number of Fortune 500 corporations. The hackers remained undetected, free to
forage, for months.
The hackers infiltrated the systems by inserting malware
into routine software updates that SolarWinds sent to customers to install on
its products, which are used to monitor internal computer networks. Software
updates customarily add new features, remove bugs and boost security. But in
this instance, the hackers commandeered the process by slipping in malicious
code, creating secret portals (called “back doors”) that granted them access to
an untold bounty of government and company secrets.
The incursion became the latest — and, it appears, by far
the worst — in a string of hacks targeting the software supply chain.
Cybersecurity experts have voiced concern for years that existing defenses,
which focus on attacks against individual end users, fail to spot malware
planted in downloads from trusted software suppliers. Such attacks are
especially worrisome because of their ability to rapidly distribute malicious
computer code to tens of thousands of unwitting customers.
This problem spurred development of a new approach, backed
by $2.2 million in federal grants and available for free, aimed at providing
end-to-end protection for the entire software supply pipeline. Named in-toto
(Latin for “as a whole”), it is the work of a team of academics led by Justin
Cappos, an associate computer science and engineering professor at New York
University. Cappos, 43, has made securing the software supply chain his life’s
work. In 2013, Popular Science named him as one of its “Brilliant Ten”
scientists under 40.
Cappos and his colleagues believe that the in-toto system,
if widely deployed, could have blocked or minimized the damage from the
SolarWinds attack. But that didn’t happen: The federal government has taken no
steps to require its software vendors, such as SolarWinds, to adopt it. Indeed,
no government agency has even inquired about it, according to Cappos.
“In security, you almost never go from making something
possible to impossible,” Cappos told ProPublica, during two video interviews
from Shanghai, where he is teaching. “You go from making it easy to making it
hard. We would have made it much harder for the [SolarWinds] attackers, and
most likely would have stopped the attack.” Although the SolarWinds breach was
a “really sneaky” approach, Cappos said, “in-toto definitely can protect
against this. It’s very possible to catch it.”
In-toto’s system has supporters among experts in the
government and corporations. When ProPublica asked Robert Beverly, who oversees
in-toto’s federal grant as a program director at the National Science
Foundation, whether using in-toto could have saved the government from the
hack, he replied, “Absolutely. There seems to be some strong evidence that had
some of the, or all of the, in-toto technologies been in place, this would have
been mitigated to some extent.” Beverly, whose NSF responsibilities include
“cybersecurity innovation for cyberinfrastructure” and who is on leave from his
post as a computer science professor at the Naval Postgraduate School, added
that it’s impossible to know for sure what impact in-toto would have had, and
that the system remains at an early stage of adoption. “Unfortunately,” said
Beverly, “it often takes some of these kinds of events to convince people to
use these kinds of technologies.”
Some companies have embraced in-toto, and others, like
Microsoft, have expressed interest. “I am a big fan of in-toto,” Kay Williams,
head of Microsoft’s initiatives in open source and supply-chain security, said
in an email to ProPublica. A second Microsoft program manager, Ralph Squillace,
praised in-toto in a recent NYU press release for applying “precisely to the
problems of supply chain confidence the community expects distributed
applications to have in the real world.” (After Williams’ initial response, Microsoft
declined to comment further.)
One senator blasted the government’s failure to use a system
it paid for. “The U.S. government invested millions of dollars in developing
technology that can protect against this threat, and while several large technology
companies have already adopted it, they are the exception,” said Sen. Ron Wyden,
D-Ore., a member of the Senate Intelligence Committee. “The government can
speed up industry adoption of this best practice by requiring every government
contractor to implement the best available technology to protect their supply
chains.”
The in-toto system requires software vendors to map out
their process for assembling computer code that will be sent to customers, and
it records what’s done at each step along the way. It then verifies
electronically that no hacker has inserted something in between steps.
Immediately before installation, a pre-installed tool automatically runs a
final check to make sure that what the customer received matches the final
product the software vendor generated for delivery, confirming that it wasn’t
tampered with in transit.
Cappos and a team of colleagues have worked to develop the
in-toto approach for years. It’s been up and running since 2018. The project
received a three-year grant from the National Science Foundation that year,
aimed at promoting “widespread practical use” of in-toto. (Later in 2018,
President Donald Trump signed the Federal Acquisition Supply Chain Security
Act, aimed at protecting government secrets from software supply-chain
threats.)
In-toto could block and reveal countless cyberattacks that currently
go undetected, according to Cappos, whose team includes Santiago Torres-Arias,
an assistant electrical and computer engineering professor at Purdue
University, and Reza Curtmola, co-director of the New Jersey Institute of
Technology’s Cybersecurity Research Center. In an August 2019 paper and
presentation to the USENIX computer conference, titled “in-toto: Providing
farm-to-table guarantees for bits and bytes,” Cappos’ team reported studying 30
major supply-chain breaches dating back to 2010. In-toto, they concluded, would
have prevented between 83% and 100% of those attacks.
“It’s available to everyone for free, paid for by the
government, and should be used by everyone,” said Cappos. “People may still be
able to break in and try to hack around it. But this is a necessary first step
and will catch a ton of these things.” The slow pace of adoption is “really
disappointing,” Cappos added. “In the long game, we’ll win. I just don’t know
that we want to go through the pain that it’ll take for everyone to wise up.”
One of in-toto’s earliest adopters, starting in 2018, was
Datadog, a SolarWinds competitor that provides monitoring software for internet
cloud applications. Now a publicly traded company with 2020 revenues of nearly
$600 million, its customers include Nasdaq, Whole Foods and Samsung. Datadog
uses in-toto to protect the security of its software updates. In an NYU press
release, Datadog staff security engineer Trishank Kuppusamy, who worked on the
program’s design and implementation, said that what distinguishes in-toto is
that it “has been designed against a very strong threat model that includes
nation-state attackers.” (Datadog did not reply to ProPublica’s requests for
comment.)
The General Services Administration, which provides access
to software for federal government agencies, still lists SolarWinds products
available for purchase. But it said in a statement that “compromised versions”
of SolarWinds programs identified by DHS are no longer available.
SolarWinds itself declined to weigh in on whether its hack
could have been prevented. “We are not going to speculate on in-toto and its
capabilities,” a spokesman said in an emailed statement. “We are focused on
protecting our customers, hardening our security and collaborating with the
industry to understand the attack and prevent similar attacks in the future.”
Previously little known to the general public, SolarWinds is
a public company based in Austin, Texas, with projected 2020 revenues of just
over $1 billion. It boasts of providing software to 320,000 customers in 199
countries, including 499 of the Fortune 500 companies. In a recent SEC filing,
the company said its flagship Orion products, the vehicle for the cyberattack,
provide about 45% of its revenues. A SolarWinds slogan: “We make IT look easy.”
After the hack was discovered, SolarWinds’ stock plunged,
and it is now facing shareholder lawsuits. The company has shifted aggressively
into damage-control mode, hiring CrowdStrike, a top cybersecurity firm; elite
Washington lobbyists; a crisis-communications advisor; and the newly formed
consulting team of Christopher Krebs, the former director of the Cybersecurity
and Infrastructure Security Agency (who was famously fired for contradicting
Trump’s claims of mass voting fraud) and Alex Stamos, former security chief at
Facebook.
News of what’s now known as the SolarWinds attack first came
on Dec. 8. That’s when FireEye, perhaps the nation’s preeminent hack-hunter,
announced that it had itself fallen victim to a “highly sophisticated state-sponsored
adversary” that had broken into its servers and stolen its “Red Team tools,”
which FireEye uses to try to hack into the computer networks of its clients as
a test of their cyber-defenses. FireEye soon discovered the attackers had
gained access through corrupted updates to the SolarWinds Orion
network-monitoring software that it used.
On the evening of Dec. 13, CISA issued an emergency
directive, identifying SolarWinds as ground zero for the hack and alerting
federal agencies using Orion products to disconnect them immediately. Over the
following weeks, investigators discovered that SolarWinds had been targeted
back in early September 2019, when hackers started testing their ability to
inject code into its software updates. After remaining undetected for months,
they inserted malware in new updates between February and June 2020. SolarWinds
estimated these infected updates affected “fewer than 18,000 of its customers.”
Precisely what the hackers saw, and stole, has yet to be
determined and is under investigation. But the full impact of the breach is
becoming clearer, as we now know it touches several tech companies, including
Microsoft. The software giant has also labored to limit the damage by helping
seize an internet domain in the U.S. that the hackers used to siphon data from
some SolarWinds customers.
Stamos told the Financial Times, in an interview after being
hired to help SolarWinds, that he believed the attackers had embedded hidden
code that would continue to give them access to companies and government
agencies for years. He compared the situation to Belgian and French farmers
going out into their fields where two world wars were fought and discovering an
“iron harvest” of unexploded ordnance each spring.
Dmitri Alperovitch, who co-founded CrowdStrike (the
cybersecurity firm SolarWinds has hired to investigate the hack) before leaving
last year to start a nonprofit policy group, said he thinks that, in theory,
the in-toto system could work. But he warned that software is so complex, with
many products and companies in the supply chain, that no one defense is a panacea.
Still, he agrees that in-toto could provide protection, and said “it’s always a
good thing to have more protection for supply chains.”
Russian intelligence services have clearly identified
supply-chain attacks “as a much better way to get in,” offering “a much bigger
set of targets,” Alperovitch said. “This is an indictment of the entire
cybersecurity industry, as well as the intelligence community, that they were
able to orchestrate such a broad, sweeping attack right under our noses.”
Click here for the
original article.