Feb 192019
 

Upgrading to Exchange 2016 CU12 may fail when using Let’s Encrypt SSL Certificates

On a Microsoft Exchange 2016 Server, utilizing Let’s Encrypt SSL Certificates, an upgrade to Cumulative Update 12 may fail. This is due to security permissions on the SSL certificate.

I later noted that this occurs on all cumulative updates when using the Let’s Encrypt SSL certificates. This includes Exchange 2016 CU13.

The CU install will fail, some services may function, but the server will not accept e-mail, or allow connections from Microsoft Outlook, or ActiveSync devices. PowerShell and EAC will not function.

The issue can be identified on this failure log:

[02/18/2019 19:24:28.0862] [2] Beginning processing Install-AuthCertificate
[02/18/2019 19:24:28.0867] [2] Ending processing Install-AuthCertificate
[02/18/2019 19:24:28.0868] [1] The following 1 error(s) occurred during task execution:
[02/18/2019 19:24:28.0868] [1] 0. ErrorRecord: Could not grant Network Service access to the certificate with thumbprint XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX because a cryptographic exception was thrown.
[02/18/2019 19:24:28.0868] [1] 0. ErrorRecord: Microsoft.Exchange.Management.SystemConfigurationTasks.AddAccessRuleCryptographicException: Could not grant Network Service access to the certificate with thumbprint XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX because a cryptographic exception was thrown. ---> System.Security.Cryptography.CryptographicException: Access is denied.
at Microsoft.Exchange.Security.Cryptography.X509Certificates.TlsCertificateInfo.CAPIAddAccessRule(X509Certificate2 certificate, AccessRule rule)
at Microsoft.Exchange.Security.Cryptography.X509Certificates.TlsCertificateInfo.AddAccessRule(X509Certificate2 certificate, AccessRule rule)
at Microsoft.Exchange.Management.SystemConfigurationTasks.ManageExchangeCertificate.EnableForServices(X509Certificate2 cert, AllowedServices services, String websiteName, Boolean requireSsl, ITopologyConfigurationSession dataSession, Server server, List`1 warningList, Boolean allowConfirmation, Boolean forceNetworkService)
--- End of inner exception stack trace ---
at Microsoft.Exchange.Configuration.Tasks.Task.ThrowError(Exception exception, ErrorCategory errorCategory, Object target, String helpUrl)
at Microsoft.Exchange.Configuration.Tasks.Task.WriteError(Exception exception, ErrorCategory category, Object target)
at Microsoft.Exchange.Management.SystemConfigurationTasks.InstallExchangeCertificate.EnableForServices(X509Certificate2 cert, AllowedServices services)
at Microsoft.Exchange.Management.SystemConfigurationTasks.InstallExchangeCertificate.InternalProcessRecord()
at Microsoft.Exchange.Configuration.Tasks.Task.b__91_1()
at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePipelineIfFailed)
[02/18/2019 19:24:28.0883] [1] [ERROR] The following error was generated when "$error.Clear();
Install-ExchangeCertificate -services "IIS, POP, IMAP" -DomainController $RoleDomainController
if ($RoleIsDatacenter -ne $true -And $RoleIsPartnerHosted -ne $true)
{
Install-AuthCertificate -DomainController $RoleDomainController
}
" was run: "Microsoft.Exchange.Management.SystemConfigurationTasks.AddAccessRuleCryptographicException: Could not grant Network Service access to the certificate with thumbprint XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX because a cryptographic exception was thrown. ---> System.Security.Cryptography.CryptographicException: Access is denied.
at Microsoft.Exchange.Security.Cryptography.X509Certificates.TlsCertificateInfo.CAPIAddAccessRule(X509Certificate2 certificate, AccessRule rule)
at Microsoft.Exchange.Security.Cryptography.X509Certificates.TlsCertificateInfo.AddAccessRule(X509Certificate2 certificate, AccessRule rule)
at Microsoft.Exchange.Management.SystemConfigurationTasks.ManageExchangeCertificate.EnableForServices(X509Certificate2 cert, AllowedServices services, String websiteName, Boolean requireSsl, ITopologyConfigurationSession dataSession, Server server, List`1 warningList, Boolean allowConfirmation, Boolean forceNetworkService)
--- End of inner exception stack trace ---
at Microsoft.Exchange.Configuration.Tasks.Task.ThrowError(Exception exception, ErrorCategory errorCategory, Object target, String helpUrl)
at Microsoft.Exchange.Configuration.Tasks.Task.WriteError(Exception exception, ErrorCategory category, Object target)
at Microsoft.Exchange.Management.SystemConfigurationTasks.InstallExchangeCertificate.EnableForServices(X509Certificate2 cert, AllowedServices services)
at Microsoft.Exchange.Management.SystemConfigurationTasks.InstallExchangeCertificate.InternalProcessRecord()
at Microsoft.Exchange.Configuration.Tasks.Task.b__91_1()
at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePipelineIfFailed)".
[02/18/2019 19:24:28.0883] [1] [ERROR] Could not grant Network Service access to the certificate with thumbprint XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX because a cryptographic exception was thrown.
[02/18/2019 19:24:28.0883] [1] [ERROR] Access is denied.
[02/18/2019 19:24:28.0883] [1] [ERROR-REFERENCE] Id=CafeComponent___ece23aa8c6744163B617570021d78090 Component=EXCHANGE14:\Current\Release\Shared\Datacenter\Setup
[02/18/2019 19:24:28.0895] [1] Setup is stopping now because of one or more critical errors.
[02/18/2019 19:24:28.0895] [1] Finished executing component tasks.
[02/18/2019 19:24:28.0925] [1] Ending processing Install-CafeRole
[02/18/2019 19:35:09.0688] [0] CurrentResult setupbase.maincore:396: 0
[02/18/2019 19:35:09.0689] [0] End of Setup

The Fix

Unfortunately because Exchange is not working, you won’t be able to use Powershell or the EAC to configure SSL certs.

To resolve this, open up the IIS Manager, right click on the Exchange Web Site, click “Edit Bindings”

IIS Exchange Edit Bindings
IIS Exchange Edit Bindings

Once the “Edit Bindings” windows is open, you’ll want to open BOTH https bindings, and click “Edit”, and then change the SSL Certificate from the Let’s Encrypt SSL cert, to the self-signed Exchange certificate that ships on the brand new install. The self-signed certification most likely will be labelled as the computer name.

Exchange SSL Bindings
Exchange SSL Bindings

If you configured the Let’s Encrypt SSL certificate on the “Exchange Backend” IIS site, you’ll also need to repeat these steps on that as well.

You can now restart the server, run the “setup.exe” on CU12 again, and it will attempt to continue and repair Exchange 2016 Cumulative Update 12.

Final Note

After the update is complete, you’ll want to restart the server. You’ll notice that the acme script, whether run automatically or manually, will not set the Let’s Encrypt certificate up again (because it’s not due for renewal). You’ll need to run the letsencrypt.exe file, and force an auto renewal which will kick off the Exchange configuration scripts (or you can manually set the certificate if you’re comfortable applying Exchange SSL certificates via PowerShell.

Oct 282018
 

I have noticed an issue when after upgrading Microsoft Exchange 2016 CU10 to Exchange 2016 CU11, services may fail to start. This issue can be intermittent, where some restarts are able to start more services, and others restarts fewer. I have observed this on 2 separate Exchange upgrades, both were CU10 to CU11.

The Problem

Recently, a customer had an issue where a Microsoft Exchange security update bricked their entire Exchange CU10 installation. Files were missing and services would not start (even after manually re-configuring all system services to their prior settings, and force starting). To fix this, we weighed our options and decided the best course of action would be to attempt the latest CU (CU11). This is because each Microsoft Exchange Cumulative update is actually a full installer that completely removes the old version, and installs the new version cleanly.

After installing CU11 we were able to rescue the Exchange installation (services could now start, and functioned), however numerous errors and warnings were now present, and we also noticed that there were some new issues with services.

One service in particular called “Net.Tcp Port Sharing Service”, would occasionally not start in time and cause all the Exchange Services not to start (Exchange is dependent on this services). Other times, this service would start, however random Exchange services would timeout.

Some of the errors and warnings included:

Event ID 7000
Source: Service Control Manager
Description:
The MSComplianceAudit service failed to start due to the following error: 
The service did not respond to the start or control request in a timely fashion.

Event ID 7009
Source: Service Control Manager
Description:
A timeout was reached (30000 milliseconds) while waiting for the MSComplianceAudit service to connect.

Event ID 7000
Source: Service Control Manager
Description:
The MSExchangeRepl service failed to start due to the following error: 
The service did not respond to the start or control request in a timely fashion.

Event ID 7009
Source: Service Control Manager
Description:
A timeout was reached (30000 milliseconds) while waiting for the MSExchangeRepl service to connect.

 

I also observed that on a few restarts, the services that failed would eventually end up restarting 10-15 minutes later (this only occurred 50% of the time).

Originally I was concerned and believed these issues were related to the original issues the customer experienced, however I upgraded my own Exchange 2016 server to CU11 and experienced the same problems (my instance was a clean fully functioning install). I also attempted to upgrade .NET to version 4.7.2 to see if this had any effect, but it did not.

When you go in to services (services.msc) and manually start the services, Exchange functions perfectly and everything works.

The Solution

As of yet, I don’t have a proper solution. I did however notice that with my customer’s environment, after it was left to sit overnight (around 8 hours), that subsequent restarts actually were able to start the majority of the services properly. It almost seemed as if it just needed time to fix itself. I’m not sure if this is because of IO load, or some type of Exchange database maintenance, but I’m waiting to see if it clears up on my instance as well after an amount of time. I’ll be keeping this post updated.

UPDATE – October 29th: I’ve confirmed for the 2nd time that the issue resolves at least 6-8 hours after the upgrade. At the end of the day I restarted my machine and everything was functioning properly.

 

If you are experiencing this issue, or can make a comment on it, please leave a comment on this post!

Oct 162018
 

In this post, I’ll be going over how to add additional and/or alternative UPN suffixes to your Active Directory. I’ll also be going over why you may require this inside of your environment.

This is also a follow up post to the article here: https://www.stephenwagner.com/2016/09/23/outlook-2016-exchange-2013-password-prompts-upn-and-samaccountname-troubles/ as Microsoft has deleted the KB 243629 article which contained the original instructions.

Why

There is a number of reasons why you may want to do this:

  1. You’re migrating to a newer version of Microsoft Outlook 2016, and require the users UPN to match the users e-mail address for auto-configure to function.
  2. Your internal domain is is a “domain.local” domain, however you want users to log in with a “domain.com” domain.
  3. You are implementing a line of business application or other piece of software that requires user’s UPNs to match their e-mail addresses.
  4. You’re performing a migration.

How

Let’s get to it! Here’s how to add an alternative UPN suffix to an Active Directory domain:

  1. Log on to your domain controller.
  2. Open “Active Directory Domains and Trusts”
  3. On the left hand side of the new window, right click on “Active Directory Domains and Trusts”, and select “Properties” (as shown below).

    Active Directory Domains and Trusts Window

    Active Directory Domains and Trusts Window

     

  4. Type in your new domain suffix in to the “Alternative UPN suffixes” box, and then click “Add”. As shown below.

    Add Alternative UPN suffix

    Add Alternative UPN suffix

     

  5. Click “Apply” and then close out of the windows.

The new UPN suffix should be available via “Active Directory Users and Computers” and you should be able to set it to users.

You can also configure the user accounts via the Exchange Administration Center (EAC). See below for an example:

Exchange Administration Center UPN Suffix

Exchange Administration Center UPN Suffix

 

Oct 082018
 
Microsoft Windows Logo

If you are running Microsoft Windows in a domain environment with WSUS configured, you may notice that you’re not able to install some FODs (Features on Demand), or use the “Turn Windows features on or off”. This will stop you from installing things like the RSAT tools, .NET Framework, Language Speech packs, etc…

You may see “failure to download files”, “cannot download”, or errors like “0x800F0954” when running DISM to install packages.

To resolve this, you need to modify your domain’s group policy settings to allow your workstations to query Windows Update servers for additional content. The workstations will still use your WSUS server for approvals, downloads, and updates, however in the event content is not found, it will query Windows Update.

Enable download of “Optional features” directly from Windows Update

  1. Open the group policy editor on your domain
  2. Create a new GPO, or modify an existing one. Make sure it applies to the computers you’d like
  3. Navigate to “Computer Configuration”, “Policies”, “Administrative Templates”, and then “System”.
  4. Double click or open “Specify settings for optional component installation and component repair”
  5. Make sure “Never attempt to download payload from Windows Update” is NOT checked
  6. Make sure “Download repair content and optional features directly from Windows Update instead of Windows Server Update Services (WSUS)” IS checked.
  7. Wait for your GPO to update, or run “gpupdate /force” on the workstations.

Please see an example of the configuration below:

Download repair content and optional features directly from Windows Update instead of Windows Server Update Services (WSUS)

You should now be able to download/install RSAT, .NET, Speech language packs, and more!

Oct 072018
 
Microsoft Windows Logo

Just a few words of warning when upgrading your VMware vSphere Windows 10 virtual machines to Windows 10 Version 1809 (October Update). When upgrading, after the first restart, you may notice multiple BSOD (Blue Screen of Death) with the error “Driver PNP Watchdog”. This will fail the upgrade. This issue may also occur on the Windows 10 Version 1903 (May Update).

Update – November 14 2018: This issue is still occurring on upgrades using the re-released November version of the October update.

Update and Fix – November 26th 2018: A very big thank you goes out to my reader Werner, who advised that the issue only occurs if the VM is in a snapshotted state. After his comment on this post, I decided to try upgraded without the VM in a snapshot state and it worked! Thanks Werner!

When the upgrade fails, the system will re-attempt until utlimately failing and reverting to the previous version of Windows 10.

In my case, I had a successful upgrade on numerous physical workstations, and a snapshot, so I decided to uninstall both the VMware tools agent, and VMware Horizon View agent. This had no affect and the VM still wouldn’t perform an upgrade.

I’m not sure if it’s the fact that it’s a VM, the VMware tools install, or the VMware Horizon View agent install, however I highly recommend waiting to upgrade until all the bugs get sorted out.

Leave a comment if you have anything to add! 🙂

Oct 052018
 
Microsoft Windows Logo

After you upgrade Microsoft Windows 10 to version 1809 (October Update) or later, you may notice that your RSAT (Remote Server Administration Tools) have uninstalled and that you cannot download or install RSAT on the new version of Windows 10. This is because Microsoft has provided the RSAT tools as part of “Features on Demand” on Windows 10 itself. This will apply to all future Windows 10 releases.

Some of you may not be familiar with using the “Features on Demand” or “DISM” tool on Windows, so I decided to write up this little post to assist you in installing RSAT on the latest version of Windows 10.

Please Note: These instructions should also be valid for Windows 10 May 2019 Update Build 1903.

Install RSAT on Windows 10 1809 (and higher)

To install RSAT on Windows 10 version 1809, open an elevated command and run the following command:

DISM.exe /Online /add-capability /CapabilityName:Rsat.ActiveDirectory.DS-LDS.Tools~~~~0.0.1.0 /CapabilityName:Rsat.BitLocker.Recovery.Tools~~~~0.0.1.0 /CapabilityName:Rsat.CertificateServices.Tools~~~~0.0.1.0 /CapabilityName:Rsat.DHCP.Tools~~~~0.0.1.0 /CapabilityName:Rsat.Dns.Tools~~~~0.0.1.0 /CapabilityName:Rsat.FailoverCluster.Management.Tools~~~~0.0.1.0 /CapabilityName:Rsat.FileServices.Tools~~~~0.0.1.0 /CapabilityName:Rsat.GroupPolicy.Management.Tools~~~~0.0.1.0 /CapabilityName:Rsat.IPAM.Client.Tools~~~~0.0.1.0 /CapabilityName:Rsat.LLDP.Tools~~~~0.0.1.0 /CapabilityName:Rsat.NetworkController.Tools~~~~0.0.1.0 /CapabilityName:Rsat.NetworkLoadBalancing.Tools~~~~0.0.1.0 /CapabilityName:Rsat.RemoteAccess.Management.Tools~~~~0.0.1.0 /CapabilityName:Rsat.RemoteDesktop.Services.Tools~~~~0.0.1.0 /CapabilityName:Rsat.ServerManager.Tools~~~~0.0.1.0 /CapabilityName:Rsat.Shielded.VM.Tools~~~~0.0.1.0 /CapabilityName:Rsat.StorageReplica.Tools~~~~0.0.1.0 /CapabilityName:Rsat.VolumeActivation.Tools~~~~0.0.1.0 /CapabilityName:Rsat.WSUS.Tools~~~~0.0.1.0 /CapabilityName:Rsat.StorageMigrationService.Management.Tools~~~~0.0.1.0 /CapabilityName:Rsat.SystemInsights.Management.Tools~~~~0.0.1.0

*Please Note: If you are using WSUS, you may not be configured to download “optional features” from Windows Update (resulting in “cannot download”, or error “0x800F0954”). To resolve this, please follow the instructions at: https://www.stephenwagner.com/2018/10/08/enable-windows-update-features-on-demand-and-turn-windows-features-on-or-off-in-wsus-environments/

Additional Notes

You’ll notice that by using the command above, we are installing multiple “capabilities”. Below is a list of the capabilities that we install to include the full RSAT feature set:

  • Rsat.ActiveDirectory.DS-LDS.Tools~~~~0.0.1.0
  • Rsat.BitLocker.Recovery.Tools~~~~0.0.1.0
  • Rsat.CertificateServices.Tools~~~~0.0.1.0
  • Rsat.DHCP.Tools~~~~0.0.1.0
  • Rsat.Dns.Tools~~~~0.0.1.0
  • Rsat.FailoverCluster.Management.Tools~~~~0.0.1.0
  • Rsat.FileServices.Tools~~~~0.0.1.0
  • Rsat.GroupPolicy.Management.Tools~~~~0.0.1.0
  • Rsat.IPAM.Client.Tools~~~~0.0.1.0
  • Rsat.LLDP.Tools~~~~0.0.1.0
  • Rsat.NetworkController.Tools~~~~0.0.1.0
  • Rsat.NetworkLoadBalancing.Tools~~~~0.0.1.0
  • Rsat.RemoteAccess.Management.Tools~~~~0.0.1.0
  • Rsat.RemoteDesktop.Services.Tools~~~~0.0.1.0
  • Rsat.ServerManager.Tools~~~~0.0.1.0
  • Rsat.Shielded.VM.Tools~~~~0.0.1.0
  • Rsat.StorageReplica.Tools~~~~0.0.1.0
  • Rsat.VolumeActivation.Tools~~~~0.0.1.0
  • Rsat.WSUS.Tools~~~~0.0.1.0
  • Rsat.StorageMigrationService.Management.Tools~~~~0.0.1.0
  • Rsat.SystemInsights.Management.Tools~~~~0.0.1.0

For more information on this change, you can visit the following URLS:

https://www.microsoft.com/en-ca/download/details.aspx?id=45520

https://docs.microsoft.com/en-us/windows-hardware/manufacture/desktop/features-on-demand-non-language-fod#remote-server-administration-tools-rsat

https://docs.microsoft.com/en-us/windows-hardware/manufacture/desktop/features-on-demand-v2–capabilities

Sep 162018
 
Microsoft Windows Logo

I’ve noticed an issue with Microsoft Windows Server 2016, where a default install, when joined to an Active Directory Domain, will not get it’s time from the domain itself, but rather from “time.windows.com”.

I first noticed this a couple months ago when I had some time issues with one of my Server 2016 member servers. I ran “net time” which reported time from the domain controller, so I simply restarted the VM and it resolved the issue (or so I thought). I did not know there was a larger underlying issue.

While performing maintenance today, I noticed that all Windows Server 2016 VMs were getting their time from “time.windows.com”. When running “w32tm /monitor”, the hosts actually reported the PDC time sources, yet it still used the internet ntp server. I checked all my Windows Server 2012 R2 member servers and they didn’t have the issue. All workstations running Windows 10 didn’t have the issue either.

When this issue occurs, you’ll notice in the event log that the Windows Time Service actually finds your domain controllers as time sources, but then overrides it with the internet server time.windows.com for some reason. The only reference you’ll find pertaining to “time.windows.com”, will be when you run the “w32tm /query /configuration” command.

We need to change the time source from that host to the domain “NT5DS” time source. We’ll do so by resetting the configuration to default settings on the member server.

How to reset the Windows Time Service (w32tm) to default settings

PLEASE NOTE: Only run this on member servers that are experiencing this issue. Do not run this on your domain controller.

  1. Open an elevated (administrative) command prompt
  2. Run the following commands:
    net stop w32time
    w32tm /unregister
    w32tm /register
    net start w32time
  3. Restart the server (may not be needed, but is a good idea)

After doing this, when running “w32tm /query /configuration” you’ll notice the time source will now reflect “NT5DS”, and the servers should now being using your domain hierarchy time sources (domain controllers).

Sep 042018
 
Microsoft Windows Logo

Microsoft is ending extended support for Windows 7 on January 14th 2020. With Windows 7 reaching it’s end of life, I highly recommend that you start planning your upgrade from Windows 7 to Windows 10.

When support ends, no more security patches or Windows updates will be available for the product. Expect numerous zero-day exploits to be released shortly after the product reaches EoL.

 

Important Points

  • Test all your applications (line of business applications) compatibility with Windows 10 before deploying
  • Test the OS compatibility on your infrastructure (example, SBS Small Business Server requires modification to support Windows 8 and Windows 10 properly)
  • Compare man-hours and support costs for an upgrade vs the cost of new computers which come pre-installed with Windows 10

More information can be found at https://www.digitallyaccurate.com/blog/2018/09/02/microsoft-windows-7-support-ending-january-2020-windows-7-end-of-life/

Aug 212018
 
Microsoft .NET Framework

You may notice on Windows Server 2012 R2, when applying Windows Updates that one or more .NET updates may fail with error code 0x80092004. This issue may affect all, or only some of your Windows Server 2012 R2 servers.

When troubleshooting this, you may notice numerous specific errors such as “Couldn’t find the hash of component: NetFx4-PenIMC”, or errors with a CAB file. These errors will probably come from update KB4054566 and KB4340558.

The Fix

To resolve this, we are going to download the updates MSU files from the Microsoft Update Catalog, and fully uninstall, then re-install the problematic updates.

Please Note: Always make sure you have a full backup before making modifications to your servers.

Please follow the instructions below:

  1. Create a folder called “updatefix” on the root of your C drive on the server
  2. Navigate to the Windows Update catalog at: https://www.catalog.update.microsoft.com/
  3. Search for KB4054566 and download the file for “Windows Server 2012 R2”, save it to the folder you created above called “updatefix” on the root of your C Drive. There should be one file in the download.
  4. Search for KB4340558 and download the files for “Windows Server 2012 R2”, save it to the folder you created above called “updatefix” on the root of your C Drive. There should be a total of 3 files in this download.
  5. Create a folder in the “updatefix” folder called “expanded”.
  6. Open an elevated command prompt, and run the following commands to extract the updates CAB files:
    expand -f:* "C:\updatefix\windows8.1-kb4338415-x64_cc34d1c48e0cc2a92f3c340ad9a0c927eb3ec2d1.msu" C:\updatefix\expanded\
    expand -f:* "C:\updatefix\windows8.1-kb4338419-x64_4d257a38e38b6b8e3d9e4763dba2ae7506b2754d.msu" C:\updatefix\expanded\
    expand -f:* "C:\updatefix\windows8.1-kb4338424-x64_e3d28f90c6b9dd7e80217b6fb0869e7b6dfe6738.msu" C:\updatefix\expanded\
    expand -f:* "C:\updatefix\windows8.1-kb4054566-x64_e780e6efac612bd0fcaf9cccfe15d6d05c9cc419.msu" C:\updatefix\expanded\
  7. Now let’s uninstall the problematic updates. Some of these commands may fail depending on which updates you have successfully installed. Run the following commands individually to remove the updates:
    dism /online /remove-package /packagepath:C:\updatefix\expanded\Windows8.1-KB4338424-x64.cab
    dism /online /remove-package /packagepath:C:\updatefix\expanded\Windows8.1-KB4338419-x64.cab
    dism /online /remove-package /packagepath:C:\updatefix\expanded\Windows8.1-KB4338415-x64.cab
    dism /online /remove-package /packagepath:C:\updatefix\expanded\Windows8.1-KB4054566-x64.cab
  8. Reboot your server.
  9. Now let’s cleanly install the updates. All of these commands should be successful when running. Run the following commands individually to install the updates:
    dism /online /add-package /packagepath:C:\updatefix\expanded\Windows8.1-KB4054566-x64.cab
    dism /online /add-package /packagepath:C:\updatefix\expanded\Windows8.1-KB4338415-x64.cab
    dism /online /add-package /packagepath:C:\updatefix\expanded\Windows8.1-KB4338419-x64.cab
    dism /online /add-package /packagepath:C:\updatefix\expanded\Windows8.1-KB4338424-x64.cab
  10. Reboot your server.
  11. You have now fixed the issue and all updates should now be cleanly installing via Windows Updates!

Leave a comment and let me know if this worked for you!

Aug 202018
 

An all too common problem is when users report e-mail delays ranging from 5 to 15 minutes. When troubleshing these types of issues, you’ll notice this commonly occurs when receiving e-mails from organizations that use Office 365. Specifically this occurs due to greylisting.

Why does this happen

You’re organization is using greylisting on your e-mail proxy/SMTP relay to reduce spam. Greylisting temporarily rejects the first send of an e-mail and waits for the sending server to re-transmit the message. This process usually takes around 5-15 minutes to complete. Greylisting is used because spammers won’t re-transmit the message, which leads to a massive reduction of spam messages coming through.

Once the sending server retransmits, the sending server IP address is added to your firewalls “safe senders” whitelist. From this point on the IP address (or server) will not be subject to greylisting (and any subsequent e-mails).

Office 365 has hundreds, if not thousands (possibly 10’s of thousands) of servers they use to transmit e-mail. The chance of multiple e-mails being sent from a single server is very slim, therefor greylisting is applied to every IP (server) that is sending e-mail because it’s different. Each e-mail from an Office 365 user can take 5-15 minutes, since a new server is used every time.

How to resolve

You’ll need to configure and add an exception to your e-mail proxy/SMTP relay/firewall. This exception can be based off domain, DNS name of sending server, or IP address ranges.

Scroll down for instructions on how to create an exception on a Sophos UTM.

Domain Exception

If you use domain based exceptions, you’ll need to configure these manually for each sending domain that you want your firewall to skip greylist checking. This is a very manual process, which requires lots of human intervention to continuously update your greylist exception.

DNS FQDN of MX Server

This method is the easiest, however most firewall or UTM’s will now allow these types of exceptions since a number of DNS queries will be needed everytime an e-mail comes in. One DNS query on the MX record, and then another DNS query on the DNS host contained in the MX record. If you can configure this type of exception, you’ll want to configure it as below:

*-com.mail.protection.outlook.com

IP Address Range

This is the best method. To create an IP address range exception, we’ll need a copy of all the IP address ranges or IP address spaces that Office 365 uses to send mail. This list can be found at: https://docs.microsoft.com/en-us/office365/SecurityCompliance/eop/exchange-online-protection-ip-addresses.

We’ll need to create an exception that skips greylist checking on the IP addresses outlined in the above link. This will stop any greylist checking on e-mails from Office 365 servers.

In my case, I use a Sophos UTM firewall, and to create an exception I had to do the following:

  1. Log on to the Webmin interface.
  2. Select “Email Protection”, then “STMP” on the left hand side, then “Exceptions” tab at the top.

    Sophos UTM E-Mail and SMTP Exception List

    Sophos UTM E-Mail and SMTP Exception List

  3. Create a “New Exception List” and call it “Office 365 GreylistWhitelist”.
  4. Check the “Greylisting” box under “Antispam”, and then check the “For these source hosts/networks”.

    Sophos UTM SMTP Create Exception

    Sophos UTM SMTP Create Exception

  5. Click the “+” button, and call the Network Definition “Exchange365-EOP-Group”. Change the type to “Network Group”.
  6. Click the “+” button in the members section, and start adding the IP spaces. Repeat this for each IP space (in total I added 23). Each network name (IP address space) requires a unique name, I named mine “Exchange365-EOP1” through “Exchange365-EOP23”.

    Sophos UTM SMTP Configure Exception

    Sophos UTM SMTP Configure Exception

  7. Click Save on the Network Group, and click Save on the exception.
  8. Enable the Exception

    Sophos UTM SMTP Exception Rule

    Sophos UTM SMTP Exception Rule

  9. Completed! You’ve now made the exception and delays should no longer occur.