re Setting the Record Straight on Bloomberg BusinessWeek’s Erroneous Article | AWS Security Blog

["Dave Farber" <farber () gmail com>]

Worth reading carefully djf 


Begin forwarded message:

> From: John Ohno <john.ohno () gmail com>
> Date: October 6, 2018 at 3:06:55 AM GMT+9
> To: dave () farber net
> Cc: ip <ip () listbox com>
> Subject: Re: [IP] Setting the Record Straight on Bloomberg BusinessWeek’s Erroneous Article | AWS Security Blog
>
> Also worth reading is The Register's coverage, which (among other things) points out some holes that weaken Apple and 
> Amazon's apparently-strong statements: https://www.theregister.co.uk/2018/10/04/supermicro_bloomberg/
>
> Security
>
> Insider threat
>
> Decoding the Chinese Super Micro super spy-chip super-scandal: What do we know – and who is telling the truth?
>
> Who's your money on? Bloomberg's sources? Apple? Amazon? Super Micro?
>
> By Kieren McCarthy in San Francisco 4 Oct 2018 at 23:01
> 136   SHARE ▼
>
> Analysis Chinese government agents sneaked spy chips into Super Micro servers used by Amazon, Apple, the US 
> government, and about 30 other organizations, giving Beijing's snoops access to highly sensitive data, according to a 
> bombshell Bloomberg report today.
>
> The story, which has been a year in the making and covers events it says happened three years ago, had a huge impact 
> on the markets: the company at the center of the story, San Jose-based Super Micro, saw its share price drop by 
> nearly 50 per cent; likewise Apple's share price dropped by just under two per cent, and Amazon's dropped by more 
> than two per cent.
>
> But the article has been strongly denied by the three main companies involved: Apple, Amazon, and Super Micro. Each 
> has issued strong and seemingly unambiguous statements denying the existence and discovery of such chips or any 
> investigation by the US intelligence services into the surveillance implants.
>
> These statements will have gone through layers of lawyers to make sure they do not open these publicly traded 
> corporations to lawsuits and securities fraud claims down the line. Similarly, Bloomberg employs veteran reporters 
> and layers of editors, who check and refine stories, and has a zero tolerance for inaccuracies.
>
> So which is true: did the Chinese government succeed in infiltrating the hardware supply chain and install spy chips 
> in highly sensitive US systems; or did Bloomberg's journalists go too far in their assertions? We'll dig in.
>
> The report
>
> First up, the key details of the exclusive. According to the report, tiny microchips that were made to look like 
> signal conditioning couplers were added to Super Micro data center server motherboards manufactured by 
> sub-contractors based in China.
>
> Those spy chips were not on the original board designs, and were secretly added after factory bosses were pressured 
> or bribed into altering the blueprints, it is claimed. The surveillance chips, we're told, contained enough memory 
> and processing power to effectively backdoor the host systems so that outside agents could, say, meddle with the 
> servers and exfiltrate information.
>
> The Bloomberg article is not particularly technical, so a lot of us are having to guesstimate how the hack worked. 
> From what we can tell, the spy chip was designed to look like an innocuous component on the motherboard with a few 
> connector pins – just enough for power and a serial interface, perhaps. One version was sandwiched between the 
> fiberglass layers of the PCB, it is claimed.
>
> The spy chip could have been placed electrically between the baseboard management controller (BMC) and its SPI flash 
> or serial EEPROM storage containing the BMC's firmware. Thus, when the BMC fetched and executed its code from this 
> memory, the spy chip would intercept the signals and modify the bitstream to inject malicious code into the BMC 
> processor, allowing its masters to control the BMC.
>
> The BMC is a crucial component on a server motherboard. It allows administrators to remotely monitor and repair 
> machines, typically over a network, without having to find the box in a data center, physically pull it out of the 
> rack, fix it, and re-rack it. The BMC and its firmware can be told to power-cycle the server, reinstall or modify the 
> host operating system, mount additional storage containing malicious code and data, access a virtual keyboard and 
> terminal connected to the computer, and so on. If you can reach the BMC and its software, you have total control over 
> the box.
>
> With the BMC compromised, it is possible the alleged spies modified the controller's firmware and/or the host 
> operating system and software to allow attackers to connect in or allow data to flow out. We've been covering BMC 
> security issues for a while.
>
> Here is Bloomberg's layman explanation for how that snoop-chip worked: the component "manipulated the core operating 
> instructions that tell the server what to do as data move across a motherboard… this happened at a crucial moment, as 
> small bits of the operating system were being stored in the board’s temporary memory en route to the server’s central 
> processor, the CPU. The implant was placed on the board in a way that allowed it to effectively edit this information 
> queue, injecting its own code or altering the order of the instructions the CPU was meant to follow."
>
> There are a few things to bear in mind: one is that it should be possible to detect weird network traffic coming from 
> the compromised machine, and another is that modifying BMC firmware on the fly to compromise the host system is 
> non-trivial but also not impossible. Various methods are described, here.
>
> "It is technically plausible," said infosec expert and US military veteran Jake Williams in a hastily organized web 
> conference on Thursday morning. "If I wanted to do this, this is how I'd do it."
>
> The BMC would be a "great place to put it," said Williams, because the controller has access to the server's main 
> memory, allowing it to inject backdoor code into the host operating system kernel. From there, it could pull down 
> second-stage spyware and execute it, assuming this doesn't set off any firewall rules.
>
> A third thing to consider is this: if true, a lot of effort went into this surveillance operation. It's not the sort 
> of thing that would be added to any Super Micro server shipping to any old company – it would be highly targeted to 
> minimize its discovery. If you've bought Super Micro kit, it's very unlikely it has a spy chip in it, we reckon, if 
> the report is correct. Other than Apple and Amazon, the other 30 or so organizations that used allegedly compromised 
> Super Micro boxes included a major bank and government contractors.
>
> A fourth thing is this: why go to the bother of smuggling another chip on the board, when a chip already due to be 
> placed in the circuitry could be tampered with during manufacture, using bribes and pressure? Why not switch the SPI 
> flash chip with a backdoored one – one that looks identical to a legit one? Perhaps the disguised signal coupler was 
> the best way to go.
>
> And a fifth thing: the chip allegedly fits on a pencil tip. That it can intercept and rewrite data on the fly from 
> SPI flash or a serial EEPROM is not impossible. However, it has to contain enough data to replace the fetched BMC 
> firmware code, that then alters the running operating system or otherwise implements a viable backdoor. Either the 
> chip pictured in Bloomberg's article is incorrect and just an illustration, and the actual device is larger, or there 
> is state-of-the-art custom semiconductor fabrication involved here.
>
> One final point: you would expect corporations like Apple and Amazon to have in place systems that detect not only 
> unexpected network traffic, but also unexpected operating system states. It should be possible that alterations to 
> the kernel and the stack of software above it should set off alarms during or after boot.
>
> Bloomberg claims the chip was first noticed in 2015 in a third-party security audit of Super Micro servers that was 
> carried out when it was doing due diligence into a company called Elemental Technologies that it was thinking of 
> acquiring. Elemental used Super Micro's servers to do super-fast video processing.
>
> Big problem
>
> Amazon reported what it found to the authorities and, according to Bloomberg, that "sent a shudder" through the 
> intelligence community because similar motherboards were in use "in Department of Defense data centers, the CIA’s 
> drone operations, and the onboard networks of Navy warships."
>
> Around the same time, Apple also found the tiny chips, according to the report, "after detecting odd network activity 
> and firmware problems." Apple contacted the FBI and gave the agency access to the actual hardware. US intelligence 
> agencies then tracked the hardware components backwards through the supply chain, and used their various spying 
> programs to sift through intercepted communications, eventually ending up with a focus on four sub-contracting 
> factories in China.
>
> According to Bloomberg, the US intelligence agencies were then able to uncover how the seeding process worked: "Plant 
> managers were approached by people who claimed to represent Super Micro or who held positions suggesting a connection 
> to the government. The middlemen would request changes to the motherboards’ original designs, initially offering 
> bribes in conjunction with their unusual requests. If that didn’t work, they threatened factory managers with 
> inspections that could shut down their plants. Once arrangements were in place, the middlemen would organize delivery 
> of the chips to the factories."
>
> This explanation seemingly passes the sniff test: it fits what we know of US intelligence agencies investigative 
> approaches, their spy programs, and how the Chinese government works when interacting with private businesses.
>
> The report then provides various forms of circumstantial evidence that adds weight to the idea that this all happened 
> by pointing to subsequent actions of both Apple and Amazon. Apple ditched Super Micro entirely as a supplier, over 
> the course of just a few weeks, despite planning to put in a massive order for thousands of motherboards. And Amazon 
> sold off its Beijing data center to its local partner, Beijing Sinnet, for $300m.
>
>
> > On Fri, Oct 5, 2018 at 2:03 PM John Ohno <john.ohno () gmail com> wrote:
> > Security researcher TheGrugq has gathered some interesting threads on this subject: 
> > https://medium.com/@thegrugq/supply-chain-security-speculation-b7b6357a5d05
> > Supply Chain Security Speculation
> >
> > Everything thrown at the wall that seemed to stick
> >
> > Bloomberg accuses the PLA of hardware tampering supply chain attacks. If this is at all true, it is a pretty big 
> > deal. If it is completely false, it is still a pretty big deal (but thats between Bloomberg’s lawyers and 
> > SuperMicro, the company allegedly shipping the hacked server boards.) Supply chain attacks are a scary vulnerability 
> > because the root of trust has to start somewhere, and if it starts in a no-name Chinese subcontractor factory…it’s 
> > maybe not the ideal foundation. I’ve attempted to collect as much actual information as I can based on the Bloomberg 
> > statement:
> >
> > The illicit chips … were connected to the baseboard management controller
> > Before the wild speculation though, it must be mentioned that the story is short on evidence and high on flat out 
> > denials.
> >
> > Update: more evidence from an earlier Ars Technica article seems to support the Bloomberg report.
> >
> > Update: Amazon is pretty emphatic that everything Bloomberg said about them and Supermicro is wrong.
> >
> > Update: In 2016 Apple did have security issues with Supermicro, but the circumstances are far from clear. It looks 
> > like maybe Apple is bluffing Supermicro about a bad firmware, then ghosts. If they actually did find a problem, 
> > engage in a coverup, then dump the whole problem on the .gov, it explains the weird messaging going on.
> >
> > Update: Apple comes out swinging with another “nope!”
> >
> > Update: ServeTheHome has a good write up on BMCs, but I think they may be attributing too much technical coherence 
> > to the Bloomberg article. The hypothetical attack – altering the password verification routine – is not particularly 
> > practical for an attacker. A backdoor with direct memory access, and just a few operations (read, write, jump) would 
> > be simpler, more robust, and much more useful.
> >
> > Something is rotten in the state of supply chain attack reports
> >
> > All of the named companies in the report flatly deny pretty much every statement in Bloomberg’s article. These 
> > denials are not “non-denial” denials, but directly refute specific statements of fact in Bloomberg’s report, as well 
> > as explicitly denying the core premise of the supply chain attack.
> >
> > Bloomberg claims that the circa 2015 modchip, about “the size of a grain of rice,” was discovered by a third party 
> > security auditor. I can think of people who are capable of detecting this sort of modchip hack. I cannot think of a 
> > reason why a due diligence audit of a server would go down to that level.
> >
> > On the other hand, Baseboard Management Controllers (BMC) and the Intelligent Platform Management Interface (IPMI) 
> > protocol are a horrendous tire fire for cyber security. That’s why Amazon’s statement about the audit rings true to 
> > me.
> >
> > The pre-acquisition audit described four issues with a web application (not hardware or chips) that SuperMicro 
> > provides for management of their motherboards. All these findings were fully addressed before we acquired Elemental. 
> > The first two issues, which the auditor deemed as critical, related to a vulnerability in versions prior to 3.15 of 
> > this web application (our audit covered prior versions of Elemental appliances as well),
> > Auditing multiple versions of the same server is already a lot of work, scouring them for camouflaged grain of rice 
> > sized backdoors seems a little excessive. The four issues:
> >
> > Two critical issues in the BMC web server (accessible over IPMI)
> > Two non critical ones (probably about encryption or lack thereof) that were mitigated by Amazon’s planned deployment
> > These findings ring true to me, this is what a typical infosec due diligence analysis is going to do — look at the 
> > interfaces and ports, see what functionality there is, what bugs there are, and what needs to be hardened/fixed.
> >
> > Stripping the boards and hunting for tiny camouflaged rogue modchips is pretty intense for an audit. However, if the 
> > modchip was buggy and alerted the auditors to dig deeper, then it is certainly possible. Things that could tip the 
> > auditors off:
> >
> > firmware errors when reflashing the modchipped unit (checksums?)
> > unusual network traffic (e.g. beaconing) generated by the modchip
> > anything else weird and unusual that raises redflags
> > Supply chain attacks exist. Is this article accurate? It feels a little off, but I don’t know.
> >
> > What do we know?
> >
> > There’s not much we can speculate about the modchip because the Bloomberg description of whatever it does is 
> > gibberish. It is safer to simply examine what is known about Supermicro’s server boards.
> >
> > Supermicro boards have third party BMC hardware to handle IPMI
> > There are at least three hardware providers: ASPEED, ATEN, and Nuvoton
> > ASPEED and Nuvoton use AMI software. ATEN has their own software stack
> > All Supermicro IPMI controllers appear to provide an extensive range of functionality that would be useful for an 
> > attacker
> > See the full range here, but the highlights include:
> >
> > Keyboard Video Mou
> > se (KVM) over IP
> > SSH
> > Serial over LAN (SOL), and SSH over SOL
> > Web server (default login: ADMIN:ADMIN)
> > Remote power management…
> > Servers get hacked via exposed BMC without a modchip all the time, just scan for the IMPI web console and use the 
> > default password. There are other ports to check for as well:
> >
> > TCP 80, 443: web interface
> > TCP 3520, 5900: KVM access
> > TCP 623: menu access, allowing full control of the hardware
> > Good supply chain attack?
> >
> > To compromise a server with a tiny modchip, a backdoor into the BMC would be pretty good. For example, a simple ICMP 
> > shell that beaconed out and provided basic commands to interact with the system would work in many places. The 
> > modchip’s backdoor would have to be more complex if the idea was to breach a hard target, but the BMC is certainly a 
> > good place to start.
> >
> > So, what’s the deal?
> >
> > For me, Bloomberg’s article could go either way. The logic of backdooring the BMC makes a lot of sense. Whether it 
> > happened in this case — given all the categorical denials — I have no idea.
> >
> > The real takeaway from this is that IPMI is a raging tire fire, BMCs are Satan spawn, and never ever expose IPMI 
> > interfaces to the Internet. Unless you want hackers, because that’s how you get hackers.
> >
> > Security
> > Infosec
> > Operational Security
> > China
> > Supply Chain
> >
> >
> > the grugq
> >
> > Information Security Researcher :: keybase.io/grugq :: https://www.patreon.com/grugq
> >
> >
> >
> > > On Thu, Oct 4, 2018 at 10:36 PM Dave Farber <farber () gmail com> wrote:
> > >
> > > > https://aws.amazon.com/blogs/security/setting-the-record-straight-on-bloomberg-businessweeks-erroneous-article/
> > > >
> > > > Setting the Record Straight on Bloomberg BusinessWeek’s Erroneous Article
> > > > 04 OCT 2018
> > > > Today, Bloomberg BusinessWeek published a story claiming that AWS was aware of modified hardware or malicious 
> > > > chips in SuperMicro motherboards in Elemental Media’s hardware at the time Amazon acquired Elemental in 2015, and 
> > > > that Amazon was aware of modified hardware or chips in AWS’s China Region.
> > > >
> > > > As we shared with Bloomberg BusinessWeek multiple times over the last couple months, this is untrue. At no time, 
> > > > past or present, have we ever found any issues relating to modified hardware or malicious chips in SuperMicro 
> > > > motherboards in any Elemental or Amazon systems. Nor have we engaged in an investigation with the government.
> > > >
> > > > There are so many inaccuracies in ‎this article as it relates to Amazon that they’re hard to count. We will name 
> > > > only a few of them here. First, when Amazon was considering acquiring Elemental, we did a lot of due diligence 
> > > > with our own security team, and also commissioned a single external security company to do a security assessment 
> > > > for us as well. That report did not identify any issues with modified chips or hardware. As is typical with most 
> > > > of these audits, it offered some recommended areas to remediate, and we fixed all critical issues before the 
> > > > acquisition closed. This was the sole external security report commissioned. Bloomberg has admittedly never seen 
> > > > our commissioned security report nor any other (and refused to share any details of any purported other report 
> > > > with us).
> > > >
> > > > The article also claims that after learning of hardware modifications and malicious chips in Elemental servers, we 
> > > > conducted a network-wide audit of SuperMicro motherboards and discovered the malicious chips in a Beijing data 
> > > > center. This claim is similarly untrue. The first and most obvious reason is that we never found modified hardware 
> > > > or malicious chips in Elemental servers. Aside from that, we never found modified hardware or malicious chips in 
> > > > servers in any of our data centers. And, this notion that we sold off the hardware and datacenter in China to our 
> > > > partner Sinnet because we wanted to rid ourselves of SuperMicro servers is absurd. Sinnet had been running these 
> > > > data centers since we ‎launched in China, they owned these data centers from the start, and the hardware we “sold” 
> > > > to them was a transfer-of-assets agreement mandated by new China regulations for non-Chinese cloud providers to 
> > > > continue to operate in China.
> > > >
> > > > Amazon employs stringent security standards across our supply chain – investigating all hardware and software 
> > > > prior to going into production and performing regular security audits internally and with our supply chain 
> > > > partners. We further strengthen our security posture by implementing our own hardware designs for critical 
> > > > components such as processors, servers, storage systems, and networking equipment.
> > > >
> > > > Security will always be our top priority. AWS is trusted by many of the world’s most risk-sensitive organizations 
> > > > precisely because we have demonstrated this unwavering commitment to putting their security above all else. We are 
> > > > constantly vigilant about potential threats to our customers, and we take swift and decisive action to address 
> > > > them whenever they are identified.
> > > >
> > > > – Steve Schmidt, Chief Information Security Officer
>
> This message was sent to the list address and trashed, but can be found online.



-------------------------------------------
Archives: https://www.listbox.com/member/archive/247/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=18849915
Unsubscribe Now: 
https://www.listbox.com/unsubscribe/?member_id=18849915&id_secret=18849915-a538de84&post_id=20181005205659:B8593CC4-C902-11E8-9500-835B961E2E9A
Powered by Listbox: https://www.listbox.com