Re: [External]Re: [SECURITY] [EXT]: [SECURITY] HSTS, wildcard certs and redirects

["Boyd, Daniel" <dboyd () BERRY EDU>]

An update and a request

We’ve narrowed down the “culprits” to two servers. In a nutshell, if a user visits either of these servers, all 
subsequent visits to other sites using our wildcard certificate fail because of HSTS. The problem is, neither server 
explicitly sends a Strict-Transport-Security response in the headers, so we don’t understand how the policy gets set. 
And, both servers that cause the problem are still accessible once the policy is set.

If anyone has run into this issue before, I’d love to chat offline about how you fixed it and would be glad to report 
back to the list a sanitized remediation plan.

Dan


Daniel H. Boyd (94C)
Director of Information Security
Office for Information Technology
Information Security Advisory Group Chair
Berry College
Phone: 706-236-1750
Fax:     706-238-5824

There are two rules to follow with your account passwords:
1. NEVER SHARE YOUR PASSWORDS WITH ANYONE (EVEN OIT!!!!)
2. If unsure, consult rule #1



From: The EDUCAUSE Security Community Group Listserv <SECURITY () LISTSERV EDUCAUSE EDU> On Behalf Of Shawn, Jason
Sent: Friday, February 08, 2019 11:40 AM
To: SECURITY () LISTSERV EDUCAUSE EDU
Subject: [External]Re: [SECURITY] [EXT]: [SECURITY] HSTS, wildcard certs and redirects

Hi, Dan:

Tl;dr - End users are establishing an SSL connection with mail.office365.com which a cert that does not match 
mail.berry.edu as a result of the includeSubDomains directive in your root domain’s HSTS policy.

At the root domain level, berry.edu is asserting HSTS as a policy for all subdomains via the includeSubDomains 
directive (More info here: https://tools.ietf.org/html/rfc6797#section-11.4).

When a user visits www.berry.edu<http://www.berry.edu> Chrome will store that HSTS policy on the user’s machine if the 
domain isn’t already preloaded (More info here: https://www.chromium.org/hsts). This ensures that any HTTP request is 
automatically directed to HTTPS at the client side.

If you open your Chrome browser you can see the relevant HSTS policy via chrome://net-internals/#hsts
You can then query for berry.edu and will see some key/value pairs similar to:
dynamic_sts_domain: berry.edu
dynamic_upgrade_mode: FORCE_HTTPS
dynamic_sts_include_subdomains: true
dynamic_sts_observed: 1549540212.760436
dynamic_sts_expiry: 1581076212.760432


When the user in Chrome uses the O365 vanity url mail.berry.edu Chrome will append the request header 
“Upgrade-Insecure-Requests: 1”. This is where the User Agent (Chrome) is applying the HSTS policy locally on the user’s 
side which forces an SSL connection to https. This happens if the user goes to http://mail.berry.edu or 
https://mail.berry.edu – the former will be forced to https by the above HSTS policy as regardless.



In the DNS record for mail.berry.edu you have set the CNAME to mail.office365.com From the client side this acts as a 
‘pseudo-redirect’  resolving the mail.berry.edu hostname to O365. On the client side Chrome will attempt to establish 
an SSL connection to 0365 which will respond with the O365 SSL certificate.

This is intentional behavior to avoid DNS-based man-in-the-middle attacks amongst other reasons. The reason it may have 
seemed to crop up recently could be user’s clearing their cache, updating their browsers, etc as browser behavior is 
client side.

Take all of this advise with a grain of salt of course, but I believe this can be resolved in a couple of ways:


  1.  Reissue your DigiCert Wildcard Cert with multi-domain Common Name/SANs support and add the O365 hostname(s) in 
your DNS records. Prior to purchasing you may want to review O365 FAQs/best practices for vanity urls to OWA as they 
may not recommend this given their endpoint hostnames could in theory change without prior notice. As this would 
introduce some cost and overhead for your institution, I would try the next two options first. See: 
https://www.digicert.com/multi-domain-ssl/


  1.  Remove the CNAME DNS record and instead host mail.berry.edu locally. Then issue a 301/302 redirect at the web 
server layer. Because SSL negotiation and HSTS policies happen prior to the HTTP 301 redirect, the browser should 
ingest your *.berry.edu DigiCert Wildcard Cert first and then redirect the user to O365. You can validate this approach 
out locally by modifying your /etc/hosts on your machine. In my opinion this approach provides you with the most 
control and if there was a malicious actor against your webmail endpoint you would be able to have traceability in your 
logs.



  1.  Remove the includeSubDomains HSTS policy for the root domain and instead create subdomain endpoint specific 
policies. This may affect users due to the expiry time for the cached hsts policy for berry.edu on their local machines 
but if the HSTS policy for mail.berry.edu the client side browser should follow the CNAME DNS response first regardless 
of HTTP or HTTPS as your CNAME entry isn’t protocol specific. You can likely test this locally as well. You may want to 
implement this regardless for better control over entry/exit points throughout your ecosystem with similar benefits I 
mentioned in #2 above. Those side benefits are discussed in the RFC in section 11.4.2 
https://tools.ietf.org/html/rfc6797#section-11.4


More info:
https://decoder.link/sslchecker/mail.berry.edu/443
https://hstspreload.org/?domain=mail.berry.edu

Hope that helps a bit!

--
Jason Shawn | Senior Director DevOps  | ellucian®  | 2003 Edmund Halley Dr. Suite 500,  Reston, Virginia 20191  | 
jason.shawn () ellucian com<mailto:jason.shawn () ellucian com> | www.ellucian.com<http://www.ellucian.com/>



From: The EDUCAUSE Security Community Group Listserv <SECURITY () LISTSERV EDUCAUSE EDU<mailto:SECURITY () LISTSERV 
EDUCAUSE EDU>> on behalf of "Boyd, Daniel" <dboyd () BERRY EDU<mailto:dboyd () BERRY EDU>>
Reply-To: The EDUCAUSE Security Community Group Listserv <SECURITY () LISTSERV EDUCAUSE EDU<mailto:SECURITY () LISTSERV 
EDUCAUSE EDU>>
Date: Monday, February 4, 2019 at 10:08 AM
To: "SECURITY () LISTSERV EDUCAUSE EDU<mailto:SECURITY () LISTSERV EDUCAUSE EDU>" <SECURITY () LISTSERV EDUCAUSE 
EDU<mailto:SECURITY () LISTSERV EDUCAUSE EDU>>
Subject: [EXT]: [SECURITY] HSTS, wildcard certs and redirects

**External Email**
We are having numerous issues with what I believe to be HSTS in relation to sites where we have created a simple URL 
for a service. For example, we use Office365 and users know that “mail.berry.edu” will get them to Outlook on the 
Office365 site. Or, at least it did.

We don’t have a specific cert for “mail.berry.edu”; we use our wildcard cert. The address redirects to Office365, of 
course, but Chrome now errors with, “Your connection is not private” and the Advanced section mentions HSTS. We get 
similar results with other sites that are set up the same way.

This started about two weeks ago.

Some users have deleted the HSTS domain policies in Chrome… I’m pretty uncomfortable with this – feels like putting a 
giant set of deadbolts on a door and leaving it ajar…

I feel like I’ve missed (or am missing) something fairly critical in all of this as HSTS is not new, but has just 
suddenly become an issue.

Is anyone experiencing this? How have you addressed the issue?

Thanks in advance for any clues…

Dan

Daniel H. Boyd (94C)
Director of Information Security
Office for Information Technology
Information Security Advisory Group Chair
Berry College
Phone: 706-236-1750
Fax:     706-238-5824

There are two rules to follow with your account passwords:
1. NEVER SHARE YOUR PASSWORDS WITH ANYONE (EVEN OIT!!!!)
2. If unsure, consult rule #1