Re: OpenSSL 1.0.1 TLS/DTLS hearbeat information disclosure CVE-2014-0160

[Kurt Seifried <kseifried () redhat com>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/08/2014 01:37 PM, Yves-Alexis Perez wrote:
> On Tue, Apr 08, 2014 at 06:17:57PM +0300, Jussi Eronen wrote:
> > Hello,
> >
> > On 04/08/2014 01:05 AM, Yves-Alexis Perez wrote:
> > > On Mon, Apr 07, 2014 at 01:56:27PM -0700, Reed Loden wrote:
> > > > Was this not coordinated with the distros at all? If not,
> > > > that seems like major fail on the reporters and NCSC-FI's
> > > > part. :/
> > >
> > > There was a mail from Red Hat on monday morning (CEST) with no 
> > > detail and a CRD to april 9th. It seems OpenSSL advisory came
> > > a bit uncoordinated, actually, which (it seems) triggered the
> > > release of the heartbeat and cloudfare posts, as well as the
> > > Red Hat one here.
> >
> > We reported the issue to OpenSSL a couple of hours before the
> > advisory was published. Our plan was to start notifications to
> > distros and other vendors after discussing with OpenSSL.
> > Codenomicon did mention us as the coordinator in the original
> > text of heartbleed.com, but the current text reflects the
> > situation quite well:
> >
> > """ Who coordinates response to this vulnerability?
> >
> > NCSC-FI took up the task of reaching out to the authors of
> > OpenSSL, software, operating system and appliance vendors, which
> > were potentially affected. However, this vulnerability was found
> > and details released independently by others before this work
> > was completed. Vendors should be notifying their users and
> > service providers. Internet service providers should be notifying
> > their end users where and when potential action is required. """
>
> Thanks for the clarification. I suppose nobody knows who are those 
> “others” who released independently?
>
> I think it might help to provide a full timeline of this. Here are
> the bits I know about, feel free to complete the missing bits:
>
> Sometimes (when?)      : Neel Mehta of Google Security discovers
> the vulnerability Later (when?)          : Google Security notifies
> OpenSSL Sometimes last week    : someones (who? OpenSSL?) notifes
> CloudFlare (and maybe other vendors) Mon, 07 Apr 2014 guess : Mark
> Cox of OpenSSL (but also working at Red Hat SRT) notifies Red Hat
> and authorizes them to share details of the vulns Mon, 07 Apr 2014
> 05:56 : Huzaifa Sidhpurwala (RH) add a bug to Red Hat bugzilla Mon,
> 07 Apr 2014 06:10 : Huzaifa Sidhpurwala sends a mail to distros 
> list with no details but an offer to request them privately Mon, 07
> Apr 2014 ~15:30: NCSC-FI reports issue to OpenSSL Mon, 07 Apr 2014
> 16:53 : Fix is committed to OpenSSL git (not sure if it was public
> or private at that point) Mon, 07 Apr 2014       : someone (who?)
> releases something (what, where?) Mon, 07 Apr 2014 17:27 : OpenSSL
> releases advisory Mon, 07 Apr 2014 18:00 : CloudFlare posts blog
> entry Mon, 07 Apr 2014 19:00 : Heartbleed.com is published Wed, 09
> Apr 2014       : initial CRD
>
> At that point, we (Debian) started some kind of “public situation
> room” on #debian-security and we tried to build updates ASAP, along
> with trying to find more info on this (for example, I'm still
> unsure how easy it really is to find some valuable data in those
> 64kB of process heap memory).
>
> I have to admit the handling of that vulnerability was really not
> the best disclosure I could find, whatever Cloudfare is thinking
> about this.
>
> It seems that some people where actually knowing about this quite
> early because of their proximitity with involved projects (Google
> Security, OpenSSL project, Red Hat Security), which I consider
> pretty normal. But no effort was apparently made to coordinate
> something at that point, until crash mode was activated sometimes
> on april 7th (which might have been the best thing to do if someone
> noticed it was exploited in the wide, but since we didn't get that
> kind of information we can only speculate)

So to respond/clear up some points:

It appears Codenomicon and Google found the vulnerability
independently. Google reported it to OpenSSL. Codenomicon reported it
to NCSC-FI, I'm not sure who (Codenomicon or NCSC-FI) drove the
notification of CloudFlare/etc. and they also reported it to OpenSSL
(I don't know if that was before or after notifying OpenSSL).

1) Mark J. Cox did not give Red Hat any advanced warning, he strongly
separates what he does with OpenSSL with what he does with Red Hat
(this is quite common at Red Hat, for example we have a guy on the
Debian security team, the Samba group, etc.). I for example sometimes
issue private CVE's in advance, but they don't get bugs filed/etc
until they hit a "public" source like distros@ or oss-security@.

2) Mark informed Red Hat and as you can see from the public time line
Huzaifa entered a bug into BZ and then notified distros@ about 14
minutes later, basically at the same time. Red Hat SRT is globally
situated so anyone from distros@ emailing us for details would have
gotten a very prompt response.

3) At this point the plan was to embargo this until April 9th (I
forget what time), giving everyone 2+ days to deal with it. So OpenSSL
in conjunction with Red Hat attempted to do a coordinated response
with the community.

4) Things blew up. My understanding is that OpenSSL made this public
due to additional reports, I suspect it boiled down to "Group A found
this flaw, reported it, and has a reproducer, and now Group B found
the same thing independently and also has a reproducer. chances are
the bad guys do as well so better to let everyone know the barn door
is open now rather than wait 2 more days" but there may be other
factors I'm not aware.

5) Monday morning: everyone is scrambling to get patches out and
update systems.

6) At least one vendor (CloudFlare) posts a blog entry stating they
were notified a week ago by Codenomicon/NCSC-FI , and claiming that it
was via "responsible disclosure". Other major vendors were not
informed (e.g. Amazon:
http://aws.amazon.com/security/security-bulletins/heartbleed-bug-concern/).


7) At least one vendor (Google) found this independently and, as I
understand it, patched their own systems (which is completely
understandable).

> I don't want to point finger, but I sincerely hope the next time 
> something like that happens, coordination will be done early in
> the processus, and relevant vendors will have a chance to prepare
> themselves

As you can see above, it was attempted, but Murphy's law took over.

> with a bit more than a two-days warning (or no warning at all). And
> I do know it's not always easy to identify a relevant group of
> vendors, but even when it's too late, coordinated disclosure and
> unique/authoritative information point is really helpful for
> everyone.
>
> Regards,

- -- 
Kurt Seifried Red Hat Security Response Team (SRT)
PGP: 0x5E267993 A90B F995 7350 148F 66BF 7554 160D 4553 5E26 7993
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBAgAGBQJTRFYiAAoJEBYNRVNeJnmTsgIQANB/i6PUyGhELSG8kRPIoRtI
u1wxlG5VNzpfl5ak1IVuNqxnf7VoiOWoLCOwv33GZOaUFBVL3OEWAwSR9ZwTaVAS
YyacAn7xcmkjOssLD/JYQ6ugrlEFi8aUmc0DNXbv5GBV1WWMEGPDb4ipgZMtFhxv
T7RMOsAlFOy6Qw0vJBI3eqUhFo+hBY07m7tu5jV1x91nb8I3iw8CpyzEADcv6Jbx
DTXTshe3L8frt8zMUj8w4E/bgjJkQw16kixjV+7hN1rul9ZKYhmjO0OyUoGwXbpF
NIiJkFEoSfGId+bSxKiDqjAazQxztSBstFUSu+/knVkny5s2hm1gu+GsoYFHsESO
dC4WAGgGlOdcTfdLn2CqHG6V2mjdv6vSWKJ+fi8tqfJ8Kn+lgSSq8GnBQKQI2RW7
q5DcCazTE9peBNnMWMVb3Dpfug3P8QLTh6Du/DJIIb+p357RAzqeqAeK+doQWOWI
1HTKrgAzjRbt3AYNDXsjJAYN9aNRXi/JCKbqQpPZV892xRYVJQlzKp8AFTU4xILu
pqfp8oSAy85UavKUFFFSwWSWeJ+prbxDxBuVFlqBJOdlHS09weTI0LMz2KIsg1yc
O8DX56IEhcsgww7qM8DoDhYNf5ZteyEw82PajBbWAfIkwSb41LjbQDfOxjiAbUTC
pW7VHcATbmncJMUv9qH/
=WLgM
-----END PGP SIGNATURE-----