Feb 192019
 

Upgrading to Exchange 2016 CU12 may fail when using Let’s Encrypt SSL Certificates

On a Microsoft Exchange 2016 Server, utilizing Let’s Encrypt SSL Certificates, an upgrade to Cumulative Update 12 may fail. This is due to security permissions on the SSL certificate.

I later noticed that this occurs on all cumulative updates when using the Let’s Encrypt SSL certificates. This includes Exchange 2016 CU13 and CU14.

The CU install will fail, some services may function, but the server will not accept e-mail, or allow connections from Microsoft Outlook, or ActiveSync devices. PowerShell and EAC will not function.

The issue can be identified on this failure log:

[02/18/2019 19:24:28.0862] [2] Beginning processing Install-AuthCertificate
[02/18/2019 19:24:28.0867] [2] Ending processing Install-AuthCertificate
[02/18/2019 19:24:28.0868] [1] The following 1 error(s) occurred during task execution:
[02/18/2019 19:24:28.0868] [1] 0. ErrorRecord: Could not grant Network Service access to the certificate with thumbprint XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX because a cryptographic exception was thrown.
[02/18/2019 19:24:28.0868] [1] 0. ErrorRecord: Microsoft.Exchange.Management.SystemConfigurationTasks.AddAccessRuleCryptographicException: Could not grant Network Service access to the certificate with thumbprint XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX because a cryptographic exception was thrown. ---> System.Security.Cryptography.CryptographicException: Access is denied.
at Microsoft.Exchange.Security.Cryptography.X509Certificates.TlsCertificateInfo.CAPIAddAccessRule(X509Certificate2 certificate, AccessRule rule)
at Microsoft.Exchange.Security.Cryptography.X509Certificates.TlsCertificateInfo.AddAccessRule(X509Certificate2 certificate, AccessRule rule)
at Microsoft.Exchange.Management.SystemConfigurationTasks.ManageExchangeCertificate.EnableForServices(X509Certificate2 cert, AllowedServices services, String websiteName, Boolean requireSsl, ITopologyConfigurationSession dataSession, Server server, List`1 warningList, Boolean allowConfirmation, Boolean forceNetworkService)
--- End of inner exception stack trace ---
at Microsoft.Exchange.Configuration.Tasks.Task.ThrowError(Exception exception, ErrorCategory errorCategory, Object target, String helpUrl)
at Microsoft.Exchange.Configuration.Tasks.Task.WriteError(Exception exception, ErrorCategory category, Object target)
at Microsoft.Exchange.Management.SystemConfigurationTasks.InstallExchangeCertificate.EnableForServices(X509Certificate2 cert, AllowedServices services)
at Microsoft.Exchange.Management.SystemConfigurationTasks.InstallExchangeCertificate.InternalProcessRecord()
at Microsoft.Exchange.Configuration.Tasks.Task.b__91_1()
at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePipelineIfFailed)
[02/18/2019 19:24:28.0883] [1] [ERROR] The following error was generated when "$error.Clear();
Install-ExchangeCertificate -services "IIS, POP, IMAP" -DomainController $RoleDomainController
if ($RoleIsDatacenter -ne $true -And $RoleIsPartnerHosted -ne $true)
{
Install-AuthCertificate -DomainController $RoleDomainController
}
" was run: "Microsoft.Exchange.Management.SystemConfigurationTasks.AddAccessRuleCryptographicException: Could not grant Network Service access to the certificate with thumbprint XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX because a cryptographic exception was thrown. ---> System.Security.Cryptography.CryptographicException: Access is denied.
at Microsoft.Exchange.Security.Cryptography.X509Certificates.TlsCertificateInfo.CAPIAddAccessRule(X509Certificate2 certificate, AccessRule rule)
at Microsoft.Exchange.Security.Cryptography.X509Certificates.TlsCertificateInfo.AddAccessRule(X509Certificate2 certificate, AccessRule rule)
at Microsoft.Exchange.Management.SystemConfigurationTasks.ManageExchangeCertificate.EnableForServices(X509Certificate2 cert, AllowedServices services, String websiteName, Boolean requireSsl, ITopologyConfigurationSession dataSession, Server server, List`1 warningList, Boolean allowConfirmation, Boolean forceNetworkService)
--- End of inner exception stack trace ---
at Microsoft.Exchange.Configuration.Tasks.Task.ThrowError(Exception exception, ErrorCategory errorCategory, Object target, String helpUrl)
at Microsoft.Exchange.Configuration.Tasks.Task.WriteError(Exception exception, ErrorCategory category, Object target)
at Microsoft.Exchange.Management.SystemConfigurationTasks.InstallExchangeCertificate.EnableForServices(X509Certificate2 cert, AllowedServices services)
at Microsoft.Exchange.Management.SystemConfigurationTasks.InstallExchangeCertificate.InternalProcessRecord()
at Microsoft.Exchange.Configuration.Tasks.Task.b__91_1()
at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePipelineIfFailed)".
[02/18/2019 19:24:28.0883] [1] [ERROR] Could not grant Network Service access to the certificate with thumbprint XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX because a cryptographic exception was thrown.
[02/18/2019 19:24:28.0883] [1] [ERROR] Access is denied.
[02/18/2019 19:24:28.0883] [1] [ERROR-REFERENCE] Id=CafeComponent___ece23aa8c6744163B617570021d78090 Component=EXCHANGE14:\Current\Release\Shared\Datacenter\Setup
[02/18/2019 19:24:28.0895] [1] Setup is stopping now because of one or more critical errors.
[02/18/2019 19:24:28.0895] [1] Finished executing component tasks.
[02/18/2019 19:24:28.0925] [1] Ending processing Install-CafeRole
[02/18/2019 19:35:09.0688] [0] CurrentResult setupbase.maincore:396: 0
[02/18/2019 19:35:09.0689] [0] End of Setup

The Fix

Unfortunately because Exchange is not working, you won’t be able to use Powershell or the EAC to configure SSL certs.

To resolve this, open up the IIS Manager, right click on the Exchange Web Site, click “Edit Bindings”

IIS Exchange Edit Bindings
IIS Exchange Edit Bindings

Once the “Edit Bindings” windows is open, you’ll want to open BOTH https bindings, and click “Edit”, and then change the SSL Certificate from the Let’s Encrypt SSL cert, to the self-signed Exchange certificate that ships on the brand new install. The self-signed certification most likely will be labelled as the computer name.

Exchange SSL Bindings
Exchange SSL Bindings

If you configured the Let’s Encrypt SSL certificate on the “Exchange Backend” IIS site, you’ll also need to repeat these steps on that as well.

You can now restart the server, run the “setup.exe” on CU12 again, and it will attempt to continue and repair Exchange 2016 Cumulative Update 12.

Final Note

After the update is complete, you’ll want to restart the server. You’ll notice that the acme script, whether run automatically or manually, will not set the Let’s Encrypt certificate up again (because it’s not due for renewal). You’ll need to run the letsencrypt.exe file, and force an auto renewal which will kick off the Exchange configuration scripts (or you can manually set the certificate if you’re comfortable applying Exchange SSL certificates via PowerShell.

Additional Resources

  6 Responses to “Exchange 2016 CU12 install/upgrade fails when using Let’s Encrypt SSL Certs”

  1. Hi Stephen, i was about to upgrade our exchange 2016 environment when i came upon your article.

    Do you have any suggestions for what i can do to fix the security permissions on our Lets encrypt certs. or advice on what i can do to not encounter this issue when doing the upgrade.
    Thanks
    Aarron

  2. Hi Aarron,

    To avoid the issue, use IIS Manager to change the Exchange certs to the self signed one’s that ship standard with Exchange. Restart the host, and then proceed with the upgrade.

    After the upgrade, use the command prompt and run the acme certificate program with the force renew flag to reconfigure the Let’s Encrypt certificate.

    Cheers
    Stephen

  3. Thanks for your advice.
    Aarron

  4. Is there a fix to prevent this from breaking CU updates when using LetsEncrypt SSL cert? Or do I need to get in the habit of going back to the default SSL cert before every CU update?

  5. Hi Eric,

    I actually don’t know. Every CU I’ve done since, I perform these steps now. This morning I actually upgraded to Exchange 2016 CU14 and performed these steps to avoid issues.

    I’m hoping someone else can chime in and answer your question.

    Stephen

  6. Thank you. Ran into this problem with 2019 CU3 (oddly CU1 didn’t have the same problem when I installed it a few months ago) and your workaround got my upgrade done.

 Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

(required)

(required)