Supermicro Nine Months Later: Were Any Chinese Spy Chips Found?
April 1, 2019 marked a return to business as usual for Supermicro (OTCMKTS:SMCI).
The company’s stock, battered by a Bloomberg Businessweek story from October of the past year, had recovered. Having dropped nearly 50% in the aftermath of the report, by the end of the trading day -- the irony of the calendar date hopefully not lost -- the stock had crested US $21.90 surpassing the US $21.40 it closed at pre-publication on October 4, 2018.
Fast forward to today, nearly nine months since this incident, and has any evidence emerged that Supermicro’s supply chain was compromised? Have any ‘spy chips’ been found?
Deny, Deny, Deny
In the immediate aftermath of Bloomberg Businessweek’s story, Supermicro was left with one option: denial. Thankfully for the company, skepticism about the story’s technical viability was beginning to mount and a consensus was building that it may be more fiction than fact.
One of the more detailed debunks of the story came from Patrick Kennedy of the server, enterprise, and storage blog Serve the Home. In a post a few days after the story broke, Kennedy outlined the technical infeasibility of the story and took particular issue with the way the authors described the server motherboard’s baseboard management controller (BMC), used to allow administrators remote access to troubleshoot or reboot the server, as a backdoor via rogue chips placed beside it:
The next inaccuracy to this paragraph is the line describing BMCs as “giving them access to the most sensitive code even on machines that have crashed or are turned off.” That is not how this technology works.
Baseboard management controllers or BMCs are active on crashed or turned off servers. They allow one to, for example, power cycle servers remotely. If you read our piece Explaining the Baseboard Management Controller or BMC in Servers BMCs are superchips. They replace a physical administrator working on a server in a data center for most tasks other than physical service (e.g. changing failed hard drives.)
When a server is powered off it is not possible to access a server’s “most sensitive code.” OS boot devices are powered off. Local storage is powered off for the main server. Further encrypted sensitive code pushed from network storage is not accessible, and a BMC would not authenticate.
This line from the Bloomberg is technically inaccurate because a powered off server’s storage with its sensitive code has no power and cannot be accessed.
An audit by Nardello & Co examined Supermicro’s supply chain, in addition to production and shipped motherboards, but found no evidence of rogue hardware.
In the immediate aftermath of the story, many of Supermicro’s customers that shepherd the most sensitive data launched a PR campaign describing the audits and security checks they do before deploying a new server such as optical inspection and X-Rays. Even if rogue chips made their way onto the motherboard -- and were technically capable of carrying out the attack described -- these checks would have caught the stowaway silicon even before the server was turned on. Supermicro also pointed out that in order for this hack to work the attacker would need to have intimate 'pin-to-pin knowledge' of the design -- something that’s not likely for an outsider.
“[Our manufacturing process is] designed so that no single Supermicro employee, single team, or contractor has unrestricted access to the complete motherboard design," the company said in a statement.
In April, Bloomberg reported that certain Huawei routers and switches had a backdoor baked into their network controllers -- perhaps placed during the production process -- but that turned out to be a bug by any other name.
Moving the Supply Chain
While Bloomberg’s story might have been a whopper, it does bring up the real and present danger of supply chain security. Being exposed to China, the supply chain is vulnerable and less sophisticated attacks and exploits have happened before given this weakness.
Prior to the publication of this story, Supermicro was well aware of this looming threat and was in the process of pushing more of its supply chain over to Taiwan. The trade war only accelerated this effort. In early May, Supermicro held a groundbreaking to mark the beginning of work on the second phase of a new 800,000-square foot factory in Taoyuan, Taiwan near Taipei.
A Smoking Gun on Neither Side
All parties involved in this story have an incredible incentive to verify every aspect and every detail, no matter how minuscule. Despite Supermicro’s audit and third-party investigation, Bloomberg has yet to retract or change its story.
This could be because neither side has a smoking gun: Supermicro is confident that the attack Bloomberg described didn’t occur, but hasn’t ruled out that something similar might be feasible. In a presentation at the recent 35th Chaos Communication Congress, Security researcher Trammell Hudson was able to connect to the server motherboard’s BMC via a rogue chip and inject new data into the Non-Volatile RAM cache of the file system thus in turn was able to run a small shell script as well as run commands as root on the BMC.
Hudson noted that the BMC had “way too many privileges.” But he also noted that his attack was done in a highly controlled environment, and that pulling it off in the wild via a compromised supply chain and manufacturing process probably didn’t happen.
Supermicro’s stock opened June 10 at US $18.98