OpenSSL: bug in modular exponentiation

[Guido Vranken <guidovranken () gmail com>]

My bignum fuzzer (https://github.com/guidovranken/bignum-fuzzer)
running on Google's oss-fuzz recently found a bug in affecting
constant-time modular exponentiation.
OpenSSL does not treat this as a security vulnerability. This is a
heads-up to developers who rely on the affected code so they can
review the impact on their applications on a case-by-case basis.

The bug is located in a function written in assembly language and the
bug can only manifest on specific processors, most likely the same as
CVE-2017-3738 (see https://www.openssl.org/news/vulnerabilities.html):

"This only affects processors that support the AVX2 but not ADX
extensions like Intel Haswell (4th generation)"

As far as I know BoringSSL and LibreSSL are not affected.

You can use the PoC below the line to see if your system is affected.

A system that is affected:

$ cat /proc/cpuinfo | grep "avx2\|adx" -o | sort -u
avx2
$ ./a.out
result is 0
result is 
179769313486231590772930519078902473361797697894230657273430081157732675805500963132708477322407536021120113879871393357658789768814416622492847430639474124377767893424865485276302219601246094119453082952085005768838150682342462881473913110540827237163350510684586298239947245938479716304835356083471597445343

A system that is not affected:

$ cat /proc/cpuinfo | grep "avx2\|adx" -o | sort -u
adx
avx2
$ ./a.out
result is 0
result is 0

-------------------

#include <openssl/bn.h>

static void do_mod_exp(int consttime)
{
    BIGNUM *res, *A = NULL, *B = NULL, *C = NULL;
    BN_CTX *ctx = BN_CTX_new();
    char* bn_str = NULL;

    res = BN_new();
    BN_dec2bn(&A,
"000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000179769313486231590772930519078902473361797697894230657273430081157732675805500963132708477322407536021120113879871393357658789768814416622492847430639474124377767893424865485276302219601246094119453082952085005768838150682342462881473913110540827237163350510684586298239947245938479716304835356083471597445343");
    BN_dec2bn(&B,
"000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000022222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222");
    BN_dec2bn(&C,
"000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000179769313486231590772930519078902473361797697894230657273430081157732675805500963132708477322407536021120113879871393357658789768814416622492847430639474124377767893424865485276302219601246094119453082952085005768838150682342462881473913110540827237163350510684586298239947245938479716304835356083471597445343");

    if ( consttime ) {
        BN_set_flags(A, BN_FLG_CONSTTIME);
    }
    BN_mod_exp(res, A, B, C, ctx);
    bn_str = BN_bn2dec(res);
    printf("result is %s\n", bn_str);
    OPENSSL_free(bn_str);
    BN_CTX_free(ctx);
    BN_free(A);
    BN_free(B);
    BN_free(C);
}

int main(void)
{
    do_mod_exp(0);
    do_mod_exp(1);
    return 0;
}