Educause Security Discussion mailing list archives
Re: [External]Re: [SECURITY] [EXT]: [SECURITY] HSTS, wildcard certs and redirects
From: "Shawn, Jason" <Jason.Shawn () ELLUCIAN COM>
Date: Wed, 13 Feb 2019 02:22:02 +0000
Hi Dan, hoping this helps you out. I suspect the STS header is set on either the IIS server side or is within the web application framework code itself. Could be set in both places too to be honest. On the IIS server side, native HSTS support is in IIS 10.0 version 1709. If you upgraded recently that configuration could be the culprit. Here’s a primer on how this is configured in IIS 10: https://docs.microsoft.com/en-us/iis/get-started/whats-new-in-iis-10-version-1709/iis-10-version-1709-hsts FYI, I validated you were running IIS and seeing the header set in the response by issuing a curl to your root domain: curl -I -L --http2 http://berry.edu HTTP/1.1 301 Moved Permanently Content-Length: 145 Content-Type: text/html; charset=UTF-8 Location: https://www.berry.edu/ Server: Microsoft-IIS/10.0 X-Powered-By: ASP.NET X-Frame-Options: ALLOW-FROM http://risevision.com Strict-Transport-Security: max-age=31536000; includeSubDomains X-Content-Type-Options: nosniff X-Xss-Protection: 1; mode=block Referrer-Policy: same-origin Date: Tue, 12 Feb 2019 23:39:26 GMT HTTP/2 200 cache-control: private content-length: 61642 content-type: text/html; charset=utf-8 server: Microsoft-IIS/10.0 x-aspnet-version: 4.0.30319 x-powered-by: ASP.NET x-frame-options: ALLOW-FROM http://risevision.com strict-transport-security: max-age=31536000; includeSubDomains x-content-type-options: nosniff x-xss-protection: 1; mode=block referrer-policy: same-origin date: Tue, 12 Feb 2019 23:39:26 GMT As you can see since my curl followed the initial root domain redirect, the STS header is included in the response. That said, when I visited your site in my browser though, I noticed your announcement about a new website and saw that at least in your online notice that your digital marketing team was running an A/B test – ie. “old site” vs “new site” in tandem. So that got me thinking if perhaps as part of your new website, perhaps the framework, content management system, and/or web application was upgraded or changed and could be adding the headers. If it’s really an A/B test, that could certainly be the root cause of the inconsistency you reported. I reviewed the curl responses again and saw that your web app is running on IIS 10.x with the .NET framework of 4.x+. So there is definitely a chance that your web container is adding headers. Typically in MVC web applications, there’s a web.xml or web.config or custom code that can append response headers along with any other content. If the new site is leveraging the new version of .NET Core (2.1x) as its framework, there’s definitely a potential gotcha within that implementation. If that’s a possibility then have the developers take a look to see if their webapp template has ConfigureService() invoked with the app.useHsts() method. It may be set by default as an initial “project” template that developers would start a new site with and I believe if there is an upgrade to an existing site to 2.1 that may be now be default behavior. (More here: https://docs.microsoft.com/en-us/aspnet/core/fundamentals/startup?view=aspnetcore-2.2 –the HSTS link is buried in the doc). Now if that’s the case, an important thing to note is that the core templates many developers start with essentially have some baseline code that allows them to develop on their local machines while “using HTTPS” and this template typically has an if/then or case statement in which the current environment is “detected”. If it’s “development”, then they don’t issue the same response headers that they would in production in other words, configuring local development to not have https enabled. But the key is that the templates do this by invoking (if prod) or not invoking (if dev) via a “UseHsts()” method invocation. If you’re experiencing some inconsistencies, perhaps a configuration on a server is erroneously being set as “development” vs. “production”. Here’s a sample template code example (one of many in the .net core github): public void Configure(IApplicationBuilder app, IHostingEnvironment env) { if (env.IsDevelopment()) { app.UseDeveloperExceptionPage(); } else { app.UseExceptionHandler("/Error"); app.UseHsts(); } app.UseHttpsRedirection(); app.UseStaticFiles(); app.UseCookiePolicy(); app.UseAuthentication(); app.UseMvc(); } FYI, I could deduce your .NET framework version to some extent because “4.0.30319” is in the x-aspnet-version response header but that lacks the “revision number” so you’re using any .NET framework of 4.0+ or above. You may in the future want to obfuscate away those server descriptions as many malicious attackers will look for these signatures and run known exploits against them. As an aside, NET Core and the ASP Frameworks have some subtle differences. You can find out more about the differences in the frameworks here: https://docs.microsoft.com/en-us/aspnet/core/fundamentals/choose-aspnet-framework?view=aspnetcore-2.2#aspnet-4x I’m definitely not a Microsoft or .NET Core expert whatsoever, but taking a look at IIS server config or the .NET Core 2.1 framework would be the next steps to finding root cause. -- Jason Shawn | Senior Director DevOps | ellucian® | 2003 Edmund Halley Dr. Suite 500, Reston, Virginia 20191 | jason.shawn () ellucian com<mailto:jason.shawn () ellucian com> | www.ellucian.com<http://www.ellucian.com/> From: The EDUCAUSE Security Community Group Listserv <SECURITY () LISTSERV EDUCAUSE EDU> on behalf of "Boyd, Daniel" <dboyd () BERRY EDU> Reply-To: The EDUCAUSE Security Community Group Listserv <SECURITY () LISTSERV EDUCAUSE EDU> Date: Monday, February 11, 2019 at 3:32 PM To: "SECURITY () LISTSERV EDUCAUSE EDU" <SECURITY () LISTSERV EDUCAUSE EDU> Subject: Re: [SECURITY] [External]Re: [SECURITY] [EXT]: [SECURITY] HSTS, wildcard certs and redirects An update and a request We’ve narrowed down the “culprits” to two servers. In a nutshell, if a user visits either of these servers, all subsequent visits to other sites using our wildcard certificate fail because of HSTS. The problem is, neither server explicitly sends a Strict-Transport-Security response in the headers, so we don’t understand how the policy gets set. And, both servers that cause the problem are still accessible once the policy is set. If anyone has run into this issue before, I’d love to chat offline about how you fixed it and would be glad to report back to the list a sanitized remediation plan. Dan Daniel H. Boyd (94C) Director of Information Security Office for Information Technology Information Security Advisory Group Chair Berry College Phone: 706-236-1750 Fax: 706-238-5824 There are two rules to follow with your account passwords: 1. NEVER SHARE YOUR PASSWORDS WITH ANYONE (EVEN OIT!!!!) 2. If unsure, consult rule #1 From: The EDUCAUSE Security Community Group Listserv <SECURITY () LISTSERV EDUCAUSE EDU> On Behalf Of Shawn, Jason Sent: Friday, February 08, 2019 11:40 AM To: SECURITY () LISTSERV EDUCAUSE EDU Subject: [External]Re: [SECURITY] [EXT]: [SECURITY] HSTS, wildcard certs and redirects Hi, Dan: Tl;dr - End users are establishing an SSL connection with mail.office365.com which a cert that does not match mail.berry.edu as a result of the includeSubDomains directive in your root domain’s HSTS policy. At the root domain level, berry.edu is asserting HSTS as a policy for all subdomains via the includeSubDomains directive (More info here: https://tools.ietf.org/html/rfc6797#section-11.4<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc6797%23section-11.4&data=02%7C01%7Cjason.shawn%40ELLUCIAN.COM%7C896613e3b60d4d16c45b08d690600bd4%7Cba4f1b25f4f74403892553e24140459f%7C0%7C0%7C636855139561467684&sdata=N0SolbNTpdVLZwWTYPwvBsLHke3aC%2FHfjoPa7wlgC4Y%3D&reserved=0>). When a user visits www.berry.edu<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.berry.edu&data=02%7C01%7Cjason.shawn%40ELLUCIAN.COM%7C896613e3b60d4d16c45b08d690600bd4%7Cba4f1b25f4f74403892553e24140459f%7C0%7C0%7C636855139561477692&sdata=KbzEE6lHknmsrff4IBmMLIswwv4lNz7ImeB12RXg53U%3D&reserved=0> Chrome will store that HSTS policy on the user’s machine if the domain isn’t already preloaded (More info here: https://www.chromium.org/hsts<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.chromium.org%2Fhsts&data=02%7C01%7Cjason.shawn%40ELLUCIAN.COM%7C896613e3b60d4d16c45b08d690600bd4%7Cba4f1b25f4f74403892553e24140459f%7C0%7C0%7C636855139561477692&sdata=d9zm4QiM7VACUizzRifeMkz3%2FCwUIzYS7vWUWt7p64E%3D&reserved=0>). This ensures that any HTTP request is automatically directed to HTTPS at the client side. If you open your Chrome browser you can see the relevant HSTS policy via chrome://net-internals/#hsts You can then query for berry.edu and will see some key/value pairs similar to: dynamic_sts_domain: berry.edu dynamic_upgrade_mode: FORCE_HTTPS dynamic_sts_include_subdomains: true dynamic_sts_observed: 1549540212.760436 dynamic_sts_expiry: 1581076212.760432 When the user in Chrome uses the O365 vanity url mail.berry.edu Chrome will append the request header “Upgrade-Insecure-Requests: 1”. This is where the User Agent (Chrome) is applying the HSTS policy locally on the user’s side which forces an SSL connection to https. This happens if the user goes to http://mail.berry.edu<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.berry.edu&data=02%7C01%7Cjason.shawn%40ELLUCIAN.COM%7C896613e3b60d4d16c45b08d690600bd4%7Cba4f1b25f4f74403892553e24140459f%7C0%7C0%7C636855139561487705&sdata=3KCpv2HqHOg4ZfdbQ2FhfeHBWRI46KVKaxvs%2FoC2kE4%3D&reserved=0> or https://mail.berry.edu<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.berry.edu&data=02%7C01%7Cjason.shawn%40ELLUCIAN.COM%7C896613e3b60d4d16c45b08d690600bd4%7Cba4f1b25f4f74403892553e24140459f%7C0%7C0%7C636855139561487705&sdata=WZYwG6N5tByA0%2BwcBk9H9DR8s4yh9i1pzfyAXXDx7Kk%3D&reserved=0> – the former will be forced to https by the above HSTS policy as regardless. In the DNS record for mail.berry.edu you have set the CNAME to mail.office365.com From the client side this acts as a ‘pseudo-redirect’ resolving the mail.berry.edu hostname to O365. On the client side Chrome will attempt to establish an SSL connection to 0365 which will respond with the O365 SSL certificate. This is intentional behavior to avoid DNS-based man-in-the-middle attacks amongst other reasons. The reason it may have seemed to crop up recently could be user’s clearing their cache, updating their browsers, etc as browser behavior is client side. Take all of this advise with a grain of salt of course, but I believe this can be resolved in a couple of ways: 1. Reissue your DigiCert Wildcard Cert with multi-domain Common Name/SANs support and add the O365 hostname(s) in your DNS records. Prior to purchasing you may want to review O365 FAQs/best practices for vanity urls to OWA as they may not recommend this given their endpoint hostnames could in theory change without prior notice. As this would introduce some cost and overhead for your institution, I would try the next two options first. See: https://www.digicert.com/multi-domain-ssl/<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.digicert.com%2Fmulti-domain-ssl%2F&data=02%7C01%7Cjason.shawn%40ELLUCIAN.COM%7C896613e3b60d4d16c45b08d690600bd4%7Cba4f1b25f4f74403892553e24140459f%7C0%7C0%7C636855139561497713&sdata=xZSxwKgjEA9dGsikql91TEjW0TAhBlB7DFF6DsOU3NY%3D&reserved=0> 1. Remove the CNAME DNS record and instead host mail.berry.edu locally. Then issue a 301/302 redirect at the web server layer. Because SSL negotiation and HSTS policies happen prior to the HTTP 301 redirect, the browser should ingest your *.berry.edu DigiCert Wildcard Cert first and then redirect the user to O365. You can validate this approach out locally by modifying your /etc/hosts on your machine. In my opinion this approach provides you with the most control and if there was a malicious actor against your webmail endpoint you would be able to have traceability in your logs. 1. Remove the includeSubDomains HSTS policy for the root domain and instead create subdomain endpoint specific policies. This may affect users due to the expiry time for the cached hsts policy for berry.edu on their local machines but if the HSTS policy for mail.berry.edu the client side browser should follow the CNAME DNS response first regardless of HTTP or HTTPS as your CNAME entry isn’t protocol specific. You can likely test this locally as well. You may want to implement this regardless for better control over entry/exit points throughout your ecosystem with similar benefits I mentioned in #2 above. Those side benefits are discussed in the RFC in section 11.4.2 https://tools.ietf.org/html/rfc6797#section-11.4<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc6797%23section-11.4&data=02%7C01%7Cjason.shawn%40ELLUCIAN.COM%7C896613e3b60d4d16c45b08d690600bd4%7Cba4f1b25f4f74403892553e24140459f%7C0%7C0%7C636855139561507717&sdata=xEnMeEKvmdOcf6pTjRZk7PHz01tE4D1k19JaGAVMGLs%3D&reserved=0> More info: https://decoder.link/sslchecker/mail.berry.edu/443<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdecoder.link%2Fsslchecker%2Fmail.berry.edu%2F443&data=02%7C01%7Cjason.shawn%40ELLUCIAN.COM%7C896613e3b60d4d16c45b08d690600bd4%7Cba4f1b25f4f74403892553e24140459f%7C0%7C0%7C636855139561507717&sdata=47jldKpZKaGhmHifNdwbb%2FVjTsvqpArvuZyqd7Xw8js%3D&reserved=0> https://hstspreload.org/?domain=mail.berry.edu<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhstspreload.org%2F%3Fdomain%3Dmail.berry.edu&data=02%7C01%7Cjason.shawn%40ELLUCIAN.COM%7C896613e3b60d4d16c45b08d690600bd4%7Cba4f1b25f4f74403892553e24140459f%7C0%7C0%7C636855139561517729&sdata=YXHgUULlIjc2dQeRlprKRZomGUD%2BaILE5zp1Jypi%2FHU%3D&reserved=0> Hope that helps a bit! -- Jason Shawn | Senior Director DevOps | ellucian® | 2003 Edmund Halley Dr. Suite 500, Reston, Virginia 20191 | jason.shawn () ellucian com<mailto:jason.shawn () ellucian com> | www.ellucian.com<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ellucian.com%2F&data=02%7C01%7Cjason.shawn%40ELLUCIAN.COM%7C896613e3b60d4d16c45b08d690600bd4%7Cba4f1b25f4f74403892553e24140459f%7C0%7C0%7C636855139561517729&sdata=Qi2dmseygEQatCdO%2BAn6Xp09q8oQee2rX2eAVnMmlqk%3D&reserved=0> From: The EDUCAUSE Security Community Group Listserv <SECURITY () LISTSERV EDUCAUSE EDU<mailto:SECURITY () LISTSERV EDUCAUSE EDU>> on behalf of "Boyd, Daniel" <dboyd () BERRY EDU<mailto:dboyd () BERRY EDU>> Reply-To: The EDUCAUSE Security Community Group Listserv <SECURITY () LISTSERV EDUCAUSE EDU<mailto:SECURITY () LISTSERV EDUCAUSE EDU>> Date: Monday, February 4, 2019 at 10:08 AM To: "SECURITY () LISTSERV EDUCAUSE EDU<mailto:SECURITY () LISTSERV EDUCAUSE EDU>" <SECURITY () LISTSERV EDUCAUSE EDU<mailto:SECURITY () LISTSERV EDUCAUSE EDU>> Subject: [EXT]: [SECURITY] HSTS, wildcard certs and redirects **External Email** We are having numerous issues with what I believe to be HSTS in relation to sites where we have created a simple URL for a service. For example, we use Office365 and users know that “mail.berry.edu” will get them to Outlook on the Office365 site. Or, at least it did. We don’t have a specific cert for “mail.berry.edu”; we use our wildcard cert. The address redirects to Office365, of course, but Chrome now errors with, “Your connection is not private” and the Advanced section mentions HSTS. We get similar results with other sites that are set up the same way. This started about two weeks ago. Some users have deleted the HSTS domain policies in Chrome… I’m pretty uncomfortable with this – feels like putting a giant set of deadbolts on a door and leaving it ajar… I feel like I’ve missed (or am missing) something fairly critical in all of this as HSTS is not new, but has just suddenly become an issue. Is anyone experiencing this? How have you addressed the issue? Thanks in advance for any clues… Dan Daniel H. Boyd (94C) Director of Information Security Office for Information Technology Information Security Advisory Group Chair Berry College Phone: 706-236-1750 Fax: 706-238-5824 There are two rules to follow with your account passwords: 1. NEVER SHARE YOUR PASSWORDS WITH ANYONE (EVEN OIT!!!!) 2. If unsure, consult rule #1
Current thread:
- HSTS, wildcard certs and redirects Boyd, Daniel (Feb 04)
- Re: [EXT]: [SECURITY] HSTS, wildcard certs and redirects Shawn, Jason (Feb 08)
- Re: [External]Re: [SECURITY] [EXT]: [SECURITY] HSTS, wildcard certs and redirects Boyd, Daniel (Feb 08)
- Re: [External]Re: [SECURITY] [EXT]: [SECURITY] HSTS, wildcard certs and redirects Boyd, Daniel (Feb 11)
- Re: [External]Re: [SECURITY] [EXT]: [SECURITY] HSTS, wildcard certs and redirects Shawn, Jason (Feb 12)
- Re: [External]Re: [SECURITY] [External]Re: [SECURITY] [EXT]: [SECURITY] HSTS, wildcard certs and redirects Boyd, Daniel (Feb 13)
- Re: [EXT]: [SECURITY] HSTS, wildcard certs and redirects Shawn, Jason (Feb 08)