re Setting the Record Straight on Bloomberg BusinessWeek’s Erroneous Article | AWS Security Blog

["Dave Farber" <farber () gmail com>]

Begin forwarded message:

> From: John Ohno <john.ohno () gmail com>
> Date: October 6, 2018 at 3:03:57 AM GMT+9
> To: dave () farber net
> Cc: ip <ip () listbox com>
> Subject: Re: [IP] Setting the Record Straight on Bloomberg BusinessWeek’s Erroneous Article | AWS Security Blog
>
> Security researcher TheGrugq has gathered some interesting threads on this subject: 
> https://medium.com/@thegrugq/supply-chain-security-speculation-b7b6357a5d05
> Supply Chain Security Speculation
>
> Everything thrown at the wall that seemed to stick
>
> Bloomberg accuses the PLA of hardware tampering supply chain attacks. If this is at all true, it is a pretty big 
> deal. If it is completely false, it is still a pretty big deal (but thats between Bloomberg’s lawyers and SuperMicro, 
> the company allegedly shipping the hacked server boards.) Supply chain attacks are a scary vulnerability because the 
> root of trust has to start somewhere, and if it starts in a no-name Chinese subcontractor factory…it’s maybe not the 
> ideal foundation. I’ve attempted to collect as much actual information as I can based on the Bloomberg statement:
>
> The illicit chips … were connected to the baseboard management controller
> Before the wild speculation though, it must be mentioned that the story is short on evidence and high on flat out 
> denials.
>
> Update: more evidence from an earlier Ars Technica article seems to support the Bloomberg report.
>
> Update: Amazon is pretty emphatic that everything Bloomberg said about them and Supermicro is wrong.
>
> Update: In 2016 Apple did have security issues with Supermicro, but the circumstances are far from clear. It looks 
> like maybe Apple is bluffing Supermicro about a bad firmware, then ghosts. If they actually did find a problem, 
> engage in a coverup, then dump the whole problem on the .gov, it explains the weird messaging going on.
>
> Update: Apple comes out swinging with another “nope!”
>
> Update: ServeTheHome has a good write up on BMCs, but I think they may be attributing too much technical coherence to 
> the Bloomberg article. The hypothetical attack – altering the password verification routine – is not particularly 
> practical for an attacker. A backdoor with direct memory access, and just a few operations (read, write, jump) would 
> be simpler, more robust, and much more useful.
>
> Something is rotten in the state of supply chain attack reports
>
> All of the named companies in the report flatly deny pretty much every statement in Bloomberg’s article. These 
> denials are not “non-denial” denials, but directly refute specific statements of fact in Bloomberg’s report, as well 
> as explicitly denying the core premise of the supply chain attack.
>
> Bloomberg claims that the circa 2015 modchip, about “the size of a grain of rice,” was discovered by a third party 
> security auditor. I can think of people who are capable of detecting this sort of modchip hack. I cannot think of a 
> reason why a due diligence audit of a server would go down to that level.
>
> On the other hand, Baseboard Management Controllers (BMC) and the Intelligent Platform Management Interface (IPMI) 
> protocol are a horrendous tire fire for cyber security. That’s why Amazon’s statement about the audit rings true to 
> me.
>
> The pre-acquisition audit described four issues with a web application (not hardware or chips) that SuperMicro 
> provides for management of their motherboards. All these findings were fully addressed before we acquired Elemental. 
> The first two issues, which the auditor deemed as critical, related to a vulnerability in versions prior to 3.15 of 
> this web application (our audit covered prior versions of Elemental appliances as well),
> Auditing multiple versions of the same server is already a lot of work, scouring them for camouflaged grain of rice 
> sized backdoors seems a little excessive. The four issues:
>
> Two critical issues in the BMC web server (accessible over IPMI)
> Two non critical ones (probably about encryption or lack thereof) that were mitigated by Amazon’s planned deployment
> These findings ring true to me, this is what a typical infosec due diligence analysis is going to do — look at the 
> interfaces and ports, see what functionality there is, what bugs there are, and what needs to be hardened/fixed.
>
> Stripping the boards and hunting for tiny camouflaged rogue modchips is pretty intense for an audit. However, if the 
> modchip was buggy and alerted the auditors to dig deeper, then it is certainly possible. Things that could tip the 
> auditors off:
>
> firmware errors when reflashing the modchipped unit (checksums?)
> unusual network traffic (e.g. beaconing) generated by the modchip
> anything else weird and unusual that raises redflags
> Supply chain attacks exist. Is this article accurate? It feels a little off, but I don’t know.
>
> What do we know?
>
> There’s not much we can speculate about the modchip because the Bloomberg description of whatever it does is 
> gibberish. It is safer to simply examine what is known about Supermicro’s server boards.
>
> Supermicro boards have third party BMC hardware to handle IPMI
> There are at least three hardware providers: ASPEED, ATEN, and Nuvoton
> ASPEED and Nuvoton use AMI software. ATEN has their own software stack
> All Supermicro IPMI controllers appear to provide an extensive range of functionality that would be useful for an 
> attacker
> See the full range here, but the highlights include:
>
> Keyboard Video Mou
> se (KVM) over IP
> SSH
> Serial over LAN (SOL), and SSH over SOL
> Web server (default login: ADMIN:ADMIN)
> Remote power management…
> Servers get hacked via exposed BMC without a modchip all the time, just scan for the IMPI web console and use the 
> default password. There are other ports to check for as well:
>
> TCP 80, 443: web interface
> TCP 3520, 5900: KVM access
> TCP 623: menu access, allowing full control of the hardware
> Good supply chain attack?
>
> To compromise a server with a tiny modchip, a backdoor into the BMC would be pretty good. For example, a simple ICMP 
> shell that beaconed out and provided basic commands to interact with the system would work in many places. The 
> modchip’s backdoor would have to be more complex if the idea was to breach a hard target, but the BMC is certainly a 
> good place to start.
>
> So, what’s the deal?
>
> For me, Bloomberg’s article could go either way. The logic of backdooring the BMC makes a lot of sense. Whether it 
> happened in this case — given all the categorical denials — I have no idea.
>
> The real takeaway from this is that IPMI is a raging tire fire, BMCs are Satan spawn, and never ever expose IPMI 
> interfaces to the Internet. Unless you want hackers, because that’s how you get hackers.
>
> Security
> Infosec
> Operational Security
> China
> Supply Chain
>
>
> the grugq
>
> Information Security Researcher :: keybase.io/grugq :: https://www.patreon.com/grugq
>
>
>
> > On Thu, Oct 4, 2018 at 10:36 PM Dave Farber <farber () gmail com> wrote:
> >
> > > https://aws.amazon.com/blogs/security/setting-the-record-straight-on-bloomberg-businessweeks-erroneous-article/
> > >
> > > Setting the Record Straight on Bloomberg BusinessWeek’s Erroneous Article
> > > 04 OCT 2018
> > > Today, Bloomberg BusinessWeek published a story claiming that AWS was aware of modified hardware or malicious chips 
> > > in SuperMicro motherboards in Elemental Media’s hardware at the time Amazon acquired Elemental in 2015, and that 
> > > Amazon was aware of modified hardware or chips in AWS’s China Region.
> > >
> > > As we shared with Bloomberg BusinessWeek multiple times over the last couple months, this is untrue. At no time, 
> > > past or present, have we ever found any issues relating to modified hardware or malicious chips in SuperMicro 
> > > motherboards in any Elemental or Amazon systems. Nor have we engaged in an investigation with the government.
> > >
> > > There are so many inaccuracies in ‎this article as it relates to Amazon that they’re hard to count. We will name 
> > > only a few of them here. First, when Amazon was considering acquiring Elemental, we did a lot of due diligence with 
> > > our own security team, and also commissioned a single external security company to do a security assessment for us 
> > > as well. That report did not identify any issues with modified chips or hardware. As is typical with most of these 
> > > audits, it offered some recommended areas to remediate, and we fixed all critical issues before the acquisition 
> > > closed. This was the sole external security report commissioned. Bloomberg has admittedly never seen our 
> > > commissioned security report nor any other (and refused to share any details of any purported other report with us).
> > >
> > > The article also claims that after learning of hardware modifications and malicious chips in Elemental servers, we 
> > > conducted a network-wide audit of SuperMicro motherboards and discovered the malicious chips in a Beijing data 
> > > center. This claim is similarly untrue. The first and most obvious reason is that we never found modified hardware 
> > > or malicious chips in Elemental servers. Aside from that, we never found modified hardware or malicious chips in 
> > > servers in any of our data centers. And, this notion that we sold off the hardware and datacenter in China to our 
> > > partner Sinnet because we wanted to rid ourselves of SuperMicro servers is absurd. Sinnet had been running these 
> > > data centers since we ‎launched in China, they owned these data centers from the start, and the hardware we “sold” 
> > > to them was a transfer-of-assets agreement mandated by new China regulations for non-Chinese cloud providers to 
> > > continue to operate in China.
> > >
> > > Amazon employs stringent security standards across our supply chain – investigating all hardware and software prior 
> > > to going into production and performing regular security audits internally and with our supply chain partners. We 
> > > further strengthen our security posture by implementing our own hardware designs for critical components such as 
> > > processors, servers, storage systems, and networking equipment.
> > >
> > > Security will always be our top priority. AWS is trusted by many of the world’s most risk-sensitive organizations 
> > > precisely because we have demonstrated this unwavering commitment to putting their security above all else. We are 
> > > constantly vigilant about potential threats to our customers, and we take swift and decisive action to address them 
> > > whenever they are identified.
> > >
> > > – Steve Schmidt, Chief Information Security Officer
>
> This message was sent to the list address and trashed, but can be found online.



-------------------------------------------
Archives: https://www.listbox.com/member/archive/247/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=18849915
Unsubscribe Now: 
https://www.listbox.com/unsubscribe/?member_id=18849915&id_secret=18849915-a538de84&post_id=20181005144249:736BACF2-C8CE-11E8-B388-FFE600B5174E
Powered by Listbox: https://www.listbox.com