Re: [External]Re: [SECURITY] [External]Re: [SECURITY] [EXT]: [SECURITY] HSTS, wildcard certs and redirects

["Boyd, Daniel" <dboyd () BERRY EDU>]

Finally! A “eureka” moment…

Based on your information, I determined that the two servers I mentioned that caused our other sites to break due to 
HSTS we pulling their “favicon” directly from the main website, setting the HSTS policy. The reason this broke other 
sites is because (and, yes, this is screwed up…) even though we can access the primary web server as simply 
“berry.edu”, that is NOT the name of the certificate on the server… it is “www2.berry.edu” with SANs set up for other 
names. It sets the HSTS policy for www.berry.edu<http://www.berry.edu> and all subdomains. The other systems 
misinterpret (although misinterpret is not the right word… this whole setup is still a massive misconfiguration) this 
to apply to “berry.edu” and all subdomains.

We modified one of the suspect servers to locally serve the favicon and now it no longer sets the HSTS policy 
incorrectly.

Thanks so much for your insight and clear explanation of the underlying issues. Time to brush up my web protocols 
skills…

Dan


Daniel H. Boyd (94C)
Director of Information Security
Office for Information Technology
Information Security Advisory Group Chair
Berry College
Phone: 706-236-1750
Fax:     706-238-5824

There are two rules to follow with your account passwords:
1. NEVER SHARE YOUR PASSWORDS WITH ANYONE (EVEN OIT!!!!)
2. If unsure, consult rule #1



From: The EDUCAUSE Security Community Group Listserv <SECURITY () LISTSERV EDUCAUSE EDU> On Behalf Of Shawn, Jason
Sent: Tuesday, February 12, 2019 9:22 PM
To: SECURITY () LISTSERV EDUCAUSE EDU
Subject: [External]Re: [SECURITY] [External]Re: [SECURITY] [EXT]: [SECURITY] HSTS, wildcard certs and redirects

Hi Dan, hoping this helps you out.

I suspect the STS header is set on either the IIS server side or is within the web application framework code itself. 
Could be set in both places too to be honest.

On the IIS server side, native HSTS support is in IIS 10.0 version 1709. If you upgraded recently that configuration 
could be the culprit. Here’s a primer on how this is configured in IIS 10: 
https://docs.microsoft.com/en-us/iis/get-started/whats-new-in-iis-10-version-1709/iis-10-version-1709-hsts

FYI, I validated you were running IIS and seeing the header set in the response by issuing a curl to your root domain:

curl -I -L --http2 http://berry.edu
HTTP/1.1 301 Moved Permanently
Content-Length: 145
Content-Type: text/html; charset=UTF-8
Location: https://www.berry.edu/
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
X-Frame-Options: ALLOW-FROM http://risevision.com
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
Referrer-Policy: same-origin
Date: Tue, 12 Feb 2019 23:39:26 GMT

HTTP/2 200
cache-control: private
content-length: 61642
content-type: text/html; charset=utf-8
server: Microsoft-IIS/10.0
x-aspnet-version: 4.0.30319
x-powered-by: ASP.NET
x-frame-options: ALLOW-FROM http://risevision.com
strict-transport-security: max-age=31536000; includeSubDomains
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
referrer-policy: same-origin
date: Tue, 12 Feb 2019 23:39:26 GMT

As you can see since my curl followed the initial root domain redirect, the STS header is included in the response.

That said, when I visited your site in my browser though, I noticed your announcement about a new website and saw that 
at least in your online notice that your digital marketing team was running an A/B test – ie. “old site” vs “new site” 
in tandem. So that got me thinking if perhaps as part of your new website, perhaps the framework, content management 
system, and/or web application was upgraded or changed and could be adding the headers. If it’s really an A/B test, 
that could certainly be the root cause of the inconsistency you reported.

I reviewed the curl responses again and saw that your web app is running on IIS 10.x with the .NET framework of 4.x+. 
So there is definitely a chance that your web container is adding headers. Typically in MVC web applications, there’s a 
web.xml or web.config or custom code that can append response headers along with any other content.

If the new site is leveraging the new version of .NET Core (2.1x) as its framework, there’s definitely a potential 
gotcha within that implementation. If that’s a possibility then have the developers take a look to see if their webapp 
template has ConfigureService() invoked with the app.useHsts() method. It may be set by default as an initial “project” 
template that developers would start a new site with and I believe if there is an upgrade to an existing site to 2.1 
that may be now be default behavior. (More here: 
https://docs.microsoft.com/en-us/aspnet/core/fundamentals/startup?view=aspnetcore-2.2 –the HSTS link is buried in the 
doc).

Now if that’s the case, an important thing to note is that the core templates many developers start with essentially 
have some baseline code that allows them to develop on their local machines while “using HTTPS” and this template 
typically has an if/then or case statement in which the current environment is “detected”. If it’s “development”, then 
they don’t issue the same response headers that they would in production in other words, configuring local development 
to not have https enabled. But the key is that the templates do this by invoking (if prod) or not invoking (if dev) via 
a “UseHsts()” method invocation. If you’re experiencing some inconsistencies, perhaps a configuration on a server is 
erroneously being set as “development” vs. “production”.

Here’s a sample template code example (one of many in the .net core github):
public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
    if (env.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
    }
    else
    {
        app.UseExceptionHandler("/Error");
        app.UseHsts();
    }

    app.UseHttpsRedirection();
    app.UseStaticFiles();
    app.UseCookiePolicy();

    app.UseAuthentication();
    app.UseMvc();
}


FYI, I could deduce your .NET framework version to some extent because “4.0.30319” is in the x-aspnet-version response 
header but that lacks the “revision number” so you’re using any .NET framework of 4.0+ or above. You may in the future 
want to obfuscate away those server descriptions as many malicious attackers will look for these signatures and run 
known exploits against them.

As an aside, NET Core and the ASP Frameworks have some subtle differences. You can find out more about the differences 
in the frameworks here: 
https://docs.microsoft.com/en-us/aspnet/core/fundamentals/choose-aspnet-framework?view=aspnetcore-2.2#aspnet-4x

I’m definitely not a Microsoft or .NET Core expert whatsoever, but taking a look at IIS server config or the .NET Core 
2.1 framework would be the next steps to finding root cause.


--
Jason Shawn | Senior Director DevOps  | ellucian®  | 2003 Edmund Halley Dr. Suite 500,  Reston, Virginia 20191 | 
jason.shawn () ellucian com<mailto:jason.shawn () ellucian com> | www.ellucian.com<http://www.ellucian.com/>



From: The EDUCAUSE Security Community Group Listserv <SECURITY () LISTSERV EDUCAUSE EDU<mailto:SECURITY () LISTSERV 
EDUCAUSE EDU>> on behalf of "Boyd, Daniel" <dboyd () BERRY EDU<mailto:dboyd () BERRY EDU>>
Reply-To: The EDUCAUSE Security Community Group Listserv <SECURITY () LISTSERV EDUCAUSE EDU<mailto:SECURITY () LISTSERV 
EDUCAUSE EDU>>
Date: Monday, February 11, 2019 at 3:32 PM
To: "SECURITY () LISTSERV EDUCAUSE EDU<mailto:SECURITY () LISTSERV EDUCAUSE EDU>" <SECURITY () LISTSERV EDUCAUSE 
EDU<mailto:SECURITY () LISTSERV EDUCAUSE EDU>>
Subject: Re: [SECURITY] [External]Re: [SECURITY] [EXT]: [SECURITY] HSTS, wildcard certs and redirects

An update and a request

We’ve narrowed down the “culprits” to two servers. In a nutshell, if a user visits either of these servers, all 
subsequent visits to other sites using our wildcard certificate fail because of HSTS. The problem is, neither server 
explicitly sends a Strict-Transport-Security response in the headers, so we don’t understand how the policy gets set. 
And, both servers that cause the problem are still accessible once the policy is set.

If anyone has run into this issue before, I’d love to chat offline about how you fixed it and would be glad to report 
back to the list a sanitized remediation plan.

Dan


Daniel H. Boyd (94C)
Director of Information Security
Office for Information Technology
Information Security Advisory Group Chair
Berry College
Phone: 706-236-1750
Fax:     706-238-5824

There are two rules to follow with your account passwords:
1. NEVER SHARE YOUR PASSWORDS WITH ANYONE (EVEN OIT!!!!)
2. If unsure, consult rule #1



From: The EDUCAUSE Security Community Group Listserv <SECURITY () LISTSERV EDUCAUSE EDU<mailto:SECURITY () LISTSERV 
EDUCAUSE EDU>> On Behalf Of Shawn, Jason
Sent: Friday, February 08, 2019 11:40 AM
To: SECURITY () LISTSERV EDUCAUSE EDU<mailto:SECURITY () LISTSERV EDUCAUSE EDU>
Subject: [External]Re: [SECURITY] [EXT]: [SECURITY] HSTS, wildcard certs and redirects

Hi, Dan:

Tl;dr - End users are establishing an SSL connection with mail.office365.com which a cert that does not match 
mail.berry.edu as a result of the includeSubDomains directive in your root domain’s HSTS policy.

At the root domain level, berry.edu is asserting HSTS as a policy for all subdomains via the includeSubDomains 
directive (More info here: 
https://tools.ietf.org/html/rfc6797#section-11.4<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc6797%23section-11.4&data=02%7C01%7Cjason.shawn%40ELLUCIAN.COM%7C896613e3b60d4d16c45b08d690600bd4%7Cba4f1b25f4f74403892553e24140459f%7C0%7C0%7C636855139561467684&sdata=N0SolbNTpdVLZwWTYPwvBsLHke3aC%2FHfjoPa7wlgC4Y%3D&reserved=0>).

When a user visits 
www.berry.edu<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.berry.edu&data=02%7C01%7Cjason.shawn%40ELLUCIAN.COM%7C896613e3b60d4d16c45b08d690600bd4%7Cba4f1b25f4f74403892553e24140459f%7C0%7C0%7C636855139561477692&sdata=KbzEE6lHknmsrff4IBmMLIswwv4lNz7ImeB12RXg53U%3D&reserved=0>
 Chrome will store that HSTS policy on the user’s machine if the domain isn’t already preloaded (More info here: 
https://www.chromium.org/hsts<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.chromium.org%2Fhsts&data=02%7C01%7Cjason.shawn%40ELLUCIAN.COM%7C896613e3b60d4d16c45b08d690600bd4%7Cba4f1b25f4f74403892553e24140459f%7C0%7C0%7C636855139561477692&sdata=d9zm4QiM7VACUizzRifeMkz3%2FCwUIzYS7vWUWt7p64E%3D&reserved=0>).
 This ensures that any HTTP request is automatically directed to HTTPS at the client side.

If you open your Chrome browser you can see the relevant HSTS policy via chrome://net-internals/#hsts
You can then query for berry.edu and will see some key/value pairs similar to:
dynamic_sts_domain: berry.edu
dynamic_upgrade_mode: FORCE_HTTPS
dynamic_sts_include_subdomains: true
dynamic_sts_observed: 1549540212.760436
dynamic_sts_expiry: 1581076212.760432


When the user in Chrome uses the O365 vanity url mail.berry.edu Chrome will append the request header 
“Upgrade-Insecure-Requests: 1”. This is where the User Agent (Chrome) is applying the HSTS policy locally on the user’s 
side which forces an SSL connection to https. This happens if the user goes to 
http://mail.berry.edu<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.berry.edu&data=02%7C01%7Cjason.shawn%40ELLUCIAN.COM%7C896613e3b60d4d16c45b08d690600bd4%7Cba4f1b25f4f74403892553e24140459f%7C0%7C0%7C636855139561487705&sdata=3KCpv2HqHOg4ZfdbQ2FhfeHBWRI46KVKaxvs%2FoC2kE4%3D&reserved=0>
 or 
https://mail.berry.edu<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.berry.edu&data=02%7C01%7Cjason.shawn%40ELLUCIAN.COM%7C896613e3b60d4d16c45b08d690600bd4%7Cba4f1b25f4f74403892553e24140459f%7C0%7C0%7C636855139561487705&sdata=WZYwG6N5tByA0%2BwcBk9H9DR8s4yh9i1pzfyAXXDx7Kk%3D&reserved=0>
 – the former will be forced to https by the above HSTS policy as regardless.



In the DNS record for mail.berry.edu you have set the CNAME to mail.office365.com From the client side this acts as a 
‘pseudo-redirect’  resolving the mail.berry.edu hostname to O365. On the client side Chrome will attempt to establish 
an SSL connection to 0365 which will respond with the O365 SSL certificate.

This is intentional behavior to avoid DNS-based man-in-the-middle attacks amongst other reasons. The reason it may have 
seemed to crop up recently could be user’s clearing their cache, updating their browsers, etc as browser behavior is 
client side.

Take all of this advise with a grain of salt of course, but I believe this can be resolved in a couple of ways:


  1.  Reissue your DigiCert Wildcard Cert with multi-domain Common Name/SANs support and add the O365 hostname(s) in 
your DNS records. Prior to purchasing you may want to review O365 FAQs/best practices for vanity urls to OWA as they 
may not recommend this given their endpoint hostnames could in theory change without prior notice. As this would 
introduce some cost and overhead for your institution, I would try the next two options first. See: 
https://www.digicert.com/multi-domain-ssl/<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.digicert.com%2Fmulti-domain-ssl%2F&data=02%7C01%7Cjason.shawn%40ELLUCIAN.COM%7C896613e3b60d4d16c45b08d690600bd4%7Cba4f1b25f4f74403892553e24140459f%7C0%7C0%7C636855139561497713&sdata=xZSxwKgjEA9dGsikql91TEjW0TAhBlB7DFF6DsOU3NY%3D&reserved=0>


  1.  Remove the CNAME DNS record and instead host mail.berry.edu locally. Then issue a 301/302 redirect at the web 
server layer. Because SSL negotiation and HSTS policies happen prior to the HTTP 301 redirect, the browser should 
ingest your *.berry.edu DigiCert Wildcard Cert first and then redirect the user to O365. You can validate this approach 
out locally by modifying your /etc/hosts on your machine. In my opinion this approach provides you with the most 
control and if there was a malicious actor against your webmail endpoint you would be able to have traceability in your 
logs.



  1.  Remove the includeSubDomains HSTS policy for the root domain and instead create subdomain endpoint specific 
policies. This may affect users due to the expiry time for the cached hsts policy for berry.edu on their local machines 
but if the HSTS policy for mail.berry.edu the client side browser should follow the CNAME DNS response first regardless 
of HTTP or HTTPS as your CNAME entry isn’t protocol specific. You can likely test this locally as well. You may want to 
implement this regardless for better control over entry/exit points throughout your ecosystem with similar benefits I 
mentioned in #2 above. Those side benefits are discussed in the RFC in section 11.4.2 
https://tools.ietf.org/html/rfc6797#section-11.4<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc6797%23section-11.4&data=02%7C01%7Cjason.shawn%40ELLUCIAN.COM%7C896613e3b60d4d16c45b08d690600bd4%7Cba4f1b25f4f74403892553e24140459f%7C0%7C0%7C636855139561507717&sdata=xEnMeEKvmdOcf6pTjRZk7PHz01tE4D1k19JaGAVMGLs%3D&reserved=0>


More info:
https://decoder.link/sslchecker/mail.berry.edu/443<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdecoder.link%2Fsslchecker%2Fmail.berry.edu%2F443&data=02%7C01%7Cjason.shawn%40ELLUCIAN.COM%7C896613e3b60d4d16c45b08d690600bd4%7Cba4f1b25f4f74403892553e24140459f%7C0%7C0%7C636855139561507717&sdata=47jldKpZKaGhmHifNdwbb%2FVjTsvqpArvuZyqd7Xw8js%3D&reserved=0>
https://hstspreload.org/?domain=mail.berry.edu<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhstspreload.org%2F%3Fdomain%3Dmail.berry.edu&data=02%7C01%7Cjason.shawn%40ELLUCIAN.COM%7C896613e3b60d4d16c45b08d690600bd4%7Cba4f1b25f4f74403892553e24140459f%7C0%7C0%7C636855139561517729&sdata=YXHgUULlIjc2dQeRlprKRZomGUD%2BaILE5zp1Jypi%2FHU%3D&reserved=0>

Hope that helps a bit!

--
Jason Shawn | Senior Director DevOps  | ellucian®  | 2003 Edmund Halley Dr. Suite 500,  Reston, Virginia 20191  | 
jason.shawn () ellucian com<mailto:jason.shawn () ellucian com> | 
www.ellucian.com<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ellucian.com%2F&data=02%7C01%7Cjason.shawn%40ELLUCIAN.COM%7C896613e3b60d4d16c45b08d690600bd4%7Cba4f1b25f4f74403892553e24140459f%7C0%7C0%7C636855139561517729&sdata=Qi2dmseygEQatCdO%2BAn6Xp09q8oQee2rX2eAVnMmlqk%3D&reserved=0>



From: The EDUCAUSE Security Community Group Listserv <SECURITY () LISTSERV EDUCAUSE EDU<mailto:SECURITY () LISTSERV 
EDUCAUSE EDU>> on behalf of "Boyd, Daniel" <dboyd () BERRY EDU<mailto:dboyd () BERRY EDU>>
Reply-To: The EDUCAUSE Security Community Group Listserv <SECURITY () LISTSERV EDUCAUSE EDU<mailto:SECURITY () LISTSERV 
EDUCAUSE EDU>>
Date: Monday, February 4, 2019 at 10:08 AM
To: "SECURITY () LISTSERV EDUCAUSE EDU<mailto:SECURITY () LISTSERV EDUCAUSE EDU>" <SECURITY () LISTSERV EDUCAUSE 
EDU<mailto:SECURITY () LISTSERV EDUCAUSE EDU>>
Subject: [EXT]: [SECURITY] HSTS, wildcard certs and redirects

**External Email**
We are having numerous issues with what I believe to be HSTS in relation to sites where we have created a simple URL 
for a service. For example, we use Office365 and users know that “mail.berry.edu” will get them to Outlook on the 
Office365 site. Or, at least it did.

We don’t have a specific cert for “mail.berry.edu”; we use our wildcard cert. The address redirects to Office365, of 
course, but Chrome now errors with, “Your connection is not private” and the Advanced section mentions HSTS. We get 
similar results with other sites that are set up the same way.

This started about two weeks ago.

Some users have deleted the HSTS domain policies in Chrome… I’m pretty uncomfortable with this – feels like putting a 
giant set of deadbolts on a door and leaving it ajar…

I feel like I’ve missed (or am missing) something fairly critical in all of this as HSTS is not new, but has just 
suddenly become an issue.

Is anyone experiencing this? How have you addressed the issue?

Thanks in advance for any clues…

Dan

Daniel H. Boyd (94C)
Director of Information Security
Office for Information Technology
Information Security Advisory Group Chair
Berry College
Phone: 706-236-1750
Fax:     706-238-5824

There are two rules to follow with your account passwords:
1. NEVER SHARE YOUR PASSWORDS WITH ANYONE (EVEN OIT!!!!)
2. If unsure, consult rule #1