Oct 282018
 

I have noticed an issue when after upgrading Microsoft Exchange 2016 CU10 to Exchange 2016 CU11, services may fail to start. This issue can be intermittent, where some restarts are able to start more services, and others restarts fewer. I have observed this on 2 separate Exchange upgrades, both were CU10 to CU11.

The Problem

Recently, a customer had an issue where a Microsoft Exchange security update bricked their entire Exchange CU10 installation. Files were missing and services would not start (even after manually re-configuring all system services to their prior settings, and force starting). To fix this, we weighed our options and decided the best course of action would be to attempt the latest CU (CU11). This is because each Microsoft Exchange Cumulative update is actually a full installer that completely removes the old version, and installs the new version cleanly.

After installing CU11 we were able to rescue the Exchange installation (services could now start, and functioned), however numerous errors and warnings were now present, and we also noticed that there were some new issues with services.

One service in particular called “Net.Tcp Port Sharing Service”, would occasionally not start in time and cause all the Exchange Services not to start (Exchange is dependent on this services). Other times, this service would start, however random Exchange services would timeout.

Some of the errors and warnings included:

Event ID 7000
Source: Service Control Manager
Description:
The MSComplianceAudit service failed to start due to the following error: 
The service did not respond to the start or control request in a timely fashion.

Event ID 7009
Source: Service Control Manager
Description:
A timeout was reached (30000 milliseconds) while waiting for the MSComplianceAudit service to connect.

Event ID 7000
Source: Service Control Manager
Description:
The MSExchangeRepl service failed to start due to the following error: 
The service did not respond to the start or control request in a timely fashion.

Event ID 7009
Source: Service Control Manager
Description:
A timeout was reached (30000 milliseconds) while waiting for the MSExchangeRepl service to connect.

 

I also observed that on a few restarts, the services that failed would eventually end up restarting 10-15 minutes later (this only occurred 50% of the time).

Originally I was concerned and believed these issues were related to the original issues the customer experienced, however I upgraded my own Exchange 2016 server to CU11 and experienced the same problems (my instance was a clean fully functioning install). I also attempted to upgrade .NET to version 4.7.2 to see if this had any effect, but it did not.

When you go in to services (services.msc) and manually start the services, Exchange functions perfectly and everything works.

The Solution

As of yet, I don’t have a proper solution. I did however notice that with my customer’s environment, after it was left to sit overnight (around 8 hours), that subsequent restarts actually were able to start the majority of the services properly. It almost seemed as if it just needed time to fix itself. I’m not sure if this is because of IO load, or some type of Exchange database maintenance, but I’m waiting to see if it clears up on my instance as well after an amount of time. I’ll be keeping this post updated.

UPDATE – October 29th: I’ve confirmed for the 2nd time that the issue resolves at least 6-8 hours after the upgrade. At the end of the day I restarted my machine and everything was functioning properly.

 

If you are experiencing this issue, or can make a comment on it, please leave a comment on this post!

Oct 162018
 

In this post, I’ll be going over how to add additional and/or alternative UPN suffixes to your Active Directory. I’ll also be going over why you may require this inside of your environment.

This is also a follow up post to the article here: https://www.stephenwagner.com/2016/09/23/outlook-2016-exchange-2013-password-prompts-upn-and-samaccountname-troubles/ as Microsoft has deleted the KB 243629 article which contained the original instructions.

Why

There is a number of reasons why you may want to do this:

  1. You’re migrating to a newer version of Microsoft Outlook 2016, and require the users UPN to match the users e-mail address for auto-configure to function.
  2. Your internal domain is is a “domain.local” domain, however you want users to log in with a “domain.com” domain.
  3. You are implementing a line of business application or other piece of software that requires user’s UPNs to match their e-mail addresses.
  4. You’re performing a migration.

How

Let’s get to it! Here’s how to add an alternative UPN suffix to an Active Directory domain:

  1. Log on to your domain controller.
  2. Open “Active Directory Domains and Trusts”
  3. On the left hand side of the new window, right click on “Active Directory Domains and Trusts”, and select “Properties” (as shown below).
    Active Directory Domains and Trusts Window

    Active Directory Domains and Trusts Window

     

  4. Type in your new domain suffix in to the “Alternative UPN suffixes” box, and then click “Add”. As shown below.
    Add Alternative UPN suffix

    Add Alternative UPN suffix

     

  5. Click “Apply” and then close out of the windows.

The new UPN suffix should be available via “Active Directory Users and Computers” and you should be able to set it to users.

You can also configure the user accounts via the Exchange Administration Center (EAC). See below for an example:

Exchange Administration Center UPN Suffix

Exchange Administration Center UPN Suffix

 

Oct 082018
 
Microsoft Windows Logo

If you are running Microsoft Windows in a domain environment with WSUS configured, you may notice that you’re not able to install some FODs (Features on Demand), or use the “Turn Windows features on or off”. This will stop you from installing things like the RSAT tools, .NET Framework, Language Speech packs, etc…

You may see “failure to download files”, “cannot download”, or errors like “0x800F0954” when running DISM to install packages.

To resolve this, you need to modify your domain’s group policy settings to allow your workstations to query Windows Update servers for additional content. The workstations will still use your WSUS server for approvals, downloads, and updates, however in the event content is not found, it will query Windows Update.

Enable download of “Optional features” directly from Windows Update

  1. Open the group policy editor on your domain
  2. Create a new GPO, or modify an existing one. Make sure it applies to the computers you’d like
  3. Navigate to “Computer Configuration”, “Policies”, “Administrative Templates”, and then “System”.
  4. Double click or open “Specify settings for optional component installation and component repair”
  5. Make sure “Never attempt to download payload from Windows Update” is NOT checked
  6. Make sure “Download repair content and optional features directly from Windows Update instead of Windows Server Update Services (WSUS)” IS checked.
  7. Wait for your GPO to update, or run “gpupdate /force” on the workstations.

Please see an example of the configuration below:

Download repair content and optional features directly from Windows Update instead of Windows Server Update Services (WSUS)

You should now be able to download/install RSAT, .NET, Speech language packs, and more!

Oct 072018
 
Microsoft Windows Logo

Just a few words of warning when upgrading your VMware vSphere Windows 10 virtual machines to Windows 10 Version 1809 (October Update). When upgrading, after the first restart, you may notice multiple BSOD (Blue Screen of Death) with the error “Driver PNP Watchdog”. This will fail the upgrade.

Update – November 14 2018: This issue is still occurring on upgrades using the re-released November version of the October update.

Update and Fix – November 26th 2018: A very big thank you goes out to my reader Werner, who advised that the issue only occurs if the VM is in a snapshotted state. After his comment on this post, I decided to try upgraded without the VM in a snapshot state and it worked! Thanks Werner!

When the upgrade fails, the system will re-attempt until utlimately failing and reverting to the previous version of Windows 10.

In my case, I had a successful upgrade on numerous physical workstations, and a snapshot, so I decided to uninstall both the VMware tools agent, and VMware Horizon View agent. This had no affect and the VM still wouldn’t perform an upgrade.

I’m not sure if it’s the fact that it’s a VM, the VMware tools install, or the VMware Horizon View agent install, however I highly recommend waiting to upgrade until all the bugs get sorted out.

Leave a comment if you have anything to add! 🙂

Oct 052018
 
Microsoft Windows Logo

After you upgrade Microsoft Windows 10 to version 1809 (October Update) or later, you may notice that your RSAT (Remote Server Administration Tools) have uninstalled and that you cannot download or install RSAT on the new version of Windows 10. This is because Microsoft has provided the RSAT tools as part of “Features on Demand” on Windows 10 itself. This will apply to all future Windows 10 releases.

Some of you may not be familiar with using the “Features on Demand” or “DISM” tool on Windows, so I decided to write up this little post to assist you in installing RSAT on the latest version of Windows 10.

Install RSAT on Windows 10 1809 (and higher)

To install RSAT on Windows 10 version 1809, open an elevated command and run the following command:

DISM.exe /Online /add-capability /CapabilityName:Rsat.ActiveDirectory.DS-LDS.Tools~~~~0.0.1.0 /CapabilityName:Rsat.BitLocker.Recovery.Tools~~~~0.0.1.0 /CapabilityName:Rsat.CertificateServices.Tools~~~~0.0.1.0 /CapabilityName:Rsat.DHCP.Tools~~~~0.0.1.0 /CapabilityName:Rsat.Dns.Tools~~~~0.0.1.0 /CapabilityName:Rsat.FailoverCluster.Management.Tools~~~~0.0.1.0 /CapabilityName:Rsat.FileServices.Tools~~~~0.0.1.0 /CapabilityName:Rsat.GroupPolicy.Management.Tools~~~~0.0.1.0 /CapabilityName:Rsat.IPAM.Client.Tools~~~~0.0.1.0 /CapabilityName:Rsat.LLDP.Tools~~~~0.0.1.0 /CapabilityName:Rsat.NetworkController.Tools~~~~0.0.1.0 /CapabilityName:Rsat.NetworkLoadBalancing.Tools~~~~0.0.1.0 /CapabilityName:Rsat.RemoteAccess.Management.Tools~~~~0.0.1.0 /CapabilityName:Rsat.RemoteDesktop.Services.Tools~~~~0.0.1.0 /CapabilityName:Rsat.ServerManager.Tools~~~~0.0.1.0 /CapabilityName:Rsat.Shielded.VM.Tools~~~~0.0.1.0 /CapabilityName:Rsat.StorageReplica.Tools~~~~0.0.1.0 /CapabilityName:Rsat.VolumeActivation.Tools~~~~0.0.1.0 /CapabilityName:Rsat.WSUS.Tools~~~~0.0.1.0 /CapabilityName:Rsat.StorageMigrationService.Management.Tools~~~~0.0.1.0 /CapabilityName:Rsat.SystemInsights.Management.Tools~~~~0.0.1.0

*Please Note: If you are using WSUS, you may not be configured to download “optional features” from Windows Update (resulting in “cannot download”, or error “0x800F0954”). To resolve this, please follow the instructions at: https://www.stephenwagner.com/2018/10/08/enable-windows-update-features-on-demand-and-turn-windows-features-on-or-off-in-wsus-environments/

Additional Notes

You’ll notice that by using the command above, we are installing multiple “capabilities”. Below is a list of the capabilities that we install to include the full RSAT feature set:

  • Rsat.ActiveDirectory.DS-LDS.Tools~~~~0.0.1.0
  • Rsat.BitLocker.Recovery.Tools~~~~0.0.1.0
  • Rsat.CertificateServices.Tools~~~~0.0.1.0
  • Rsat.DHCP.Tools~~~~0.0.1.0
  • Rsat.Dns.Tools~~~~0.0.1.0
  • Rsat.FailoverCluster.Management.Tools~~~~0.0.1.0
  • Rsat.FileServices.Tools~~~~0.0.1.0
  • Rsat.GroupPolicy.Management.Tools~~~~0.0.1.0
  • Rsat.IPAM.Client.Tools~~~~0.0.1.0
  • Rsat.LLDP.Tools~~~~0.0.1.0
  • Rsat.NetworkController.Tools~~~~0.0.1.0
  • Rsat.NetworkLoadBalancing.Tools~~~~0.0.1.0
  • Rsat.RemoteAccess.Management.Tools~~~~0.0.1.0
  • Rsat.RemoteDesktop.Services.Tools~~~~0.0.1.0
  • Rsat.ServerManager.Tools~~~~0.0.1.0
  • Rsat.Shielded.VM.Tools~~~~0.0.1.0
  • Rsat.StorageReplica.Tools~~~~0.0.1.0
  • Rsat.VolumeActivation.Tools~~~~0.0.1.0
  • Rsat.WSUS.Tools~~~~0.0.1.0
  • Rsat.StorageMigrationService.Management.Tools~~~~0.0.1.0
  • Rsat.SystemInsights.Management.Tools~~~~0.0.1.0

For more information on this change, you can visit the following URLS:

https://www.microsoft.com/en-ca/download/details.aspx?id=45520

https://docs.microsoft.com/en-us/windows-hardware/manufacture/desktop/features-on-demand-non-language-fod#remote-server-administration-tools-rsat

https://docs.microsoft.com/en-us/windows-hardware/manufacture/desktop/features-on-demand-v2–capabilities

Sep 162018
 
Microsoft Windows Logo

I’ve noticed an issue with Microsoft Windows Server 2016, where a default install, when joined to an Active Directory Domain, will not get it’s time from the domain itself, but rather from “time.windows.com”.

I first noticed this a couple months ago when I had some time issues with one of my Server 2016 member servers. I ran “net time” which reported time from the domain controller, so I simply restarted the VM and it resolved the issue (or so I thought). I did not know there was a larger underlying issue.

While performing maintenance today, I noticed that all Windows Server 2016 VMs were getting their time from “time.windows.com”. When running “w32tm /monitor”, the hosts actually reported the PDC time sources, yet it still used the internet ntp server. I checked all my Windows Server 2012 R2 member servers and they didn’t have the issue. All workstations running Windows 10 didn’t have the issue either.

When this issue occurs, you’ll notice in the event log that the Windows Time Service actually finds your domain controllers as time sources, but then overrides it with the internet server time.windows.com for some reason. The only reference you’ll find pertaining to “time.windows.com”, will be when you run the “w32tm /query /configuration” command.

We need to change the time source from that host to the domain “NT5DS” time source. We’ll do so by resetting the configuration to default settings on the member server.

How to reset the Windows Time Service (w32tm) to default settings

PLEASE NOTE: Only run this on member servers that are experiencing this issue. Do not run this on your domain controller.

  1. Open an elevated (administrative) command prompt
  2. Run the following commands:
    net stop w32time
    w32tm /unregister
    w32tm /register
    net start w32time
  3. Restart the server (may not be needed, but is a good idea)

After doing this, when running “w32tm /query /configuration” you’ll notice the time source will now reflect “NT5DS”, and the servers should now being using your domain hierarchy time sources (domain controllers).

Sep 042018
 
Microsoft Windows Logo

Microsoft is ending extended support for Windows 7 on January 14th 2020. With Windows 7 reaching it’s end of life, I highly recommend that you start planning your upgrade from Windows 7 to Windows 10.

When support ends, no more security patches or Windows updates will be available for the product. Expect numerous zero-day exploits to be released shortly after the product reaches EoL.

 

Important Points

  • Test all your applications (line of business applications) compatibility with Windows 10 before deploying
  • Test the OS compatibility on your infrastructure (example, SBS Small Business Server requires modification to support Windows 8 and Windows 10 properly)
  • Compare man-hours and support costs for an upgrade vs the cost of new computers which come pre-installed with Windows 10

More information can be found at https://www.digitallyaccurate.com/blog/2018/09/02/microsoft-windows-7-support-ending-january-2020-windows-7-end-of-life/

Aug 212018
 
Microsoft .NET Framework

You may notice on Windows Server 2012 R2, when applying Windows Updates that one or more .NET updates may fail with error code 0x80092004. This issue may affect all, or only some of your Windows Server 2012 R2 servers.

When troubleshooting this, you may notice numerous specific errors such as “Couldn’t find the hash of component: NetFx4-PenIMC”, or errors with a CAB file. These errors will probably come from update KB4054566 and KB4340558.

 

The Fix

To resolve this, we are going to download the updates MSU files from the Microsoft Update Catalog, and fully uninstall, then re-install the problematic updates.

Please Note: Always make sure you have a full backup before making modifications to your servers.

Please follow the instructions below:

  1. Create a folder called “updatefix” on the root of your C drive on the server
  2. Navigate to the Windows Update catalog at: https://www.catalog.update.microsoft.com/
  3. Search for KB4054566 and download the file for “Windows Server 2012 R2”, save it to the folder you created above called “updatefix” on the root of your C Drive. There should be one file in the download.
  4. Search for KB4340558 and download the files for “Windows Server 2012 R2”, save it to the folder you created above called “updatefix” on the root of your C Drive. There should be a total of 3 files in this download.
  5. Create a folder in the “updatefix” folder called “expanded”.
  6. Open an elevated command prompt, and run the following commands to extract the updates CAB files:
    expand -f:* "C:\updatefix\windows8.1-kb4338415-x64_cc34d1c48e0cc2a92f3c340ad9a0c927eb3ec2d1.msu" C:\updatefix\expanded\
    expand -f:* "C:\updatefix\windows8.1-kb4338419-x64_4d257a38e38b6b8e3d9e4763dba2ae7506b2754d.msu" C:\updatefix\expanded\
    expand -f:* "C:\updatefix\windows8.1-kb4338424-x64_e3d28f90c6b9dd7e80217b6fb0869e7b6dfe6738.msu" C:\updatefix\expanded\
    expand -f:* "C:\updatefix\windows8.1-kb4054566-x64_e780e6efac612bd0fcaf9cccfe15d6d05c9cc419.msu" C:\updatefix\expanded\
  7. Now let’s uninstall the problematic updates. Some of these commands may fail depending on which updates you have successfully installed. Run the following commands individually to remove the updates:
    dism /online /remove-package /packagepath:C:\updatefix\expanded\Windows8.1-KB4338424-x64.cab
    dism /online /remove-package /packagepath:C:\updatefix\expanded\Windows8.1-KB4338419-x64.cab
    dism /online /remove-package /packagepath:C:\updatefix\expanded\Windows8.1-KB4338415-x64.cab
    dism /online /remove-package /packagepath:C:\updatefix\expanded\Windows8.1-KB4054566-x64.cab
  8. Now let’s cleanly install the updates. All of these commands should be successful when running. Run the following commands individually to install the updates:
    dism /online /add-package /packagepath:C:\updatefix\expanded\Windows8.1-KB4054566-x64.cab
    dism /online /add-package /packagepath:C:\updatefix\expanded\Windows8.1-KB4338415-x64.cab
    dism /online /add-package /packagepath:C:\updatefix\expanded\Windows8.1-KB4338419-x64.cab
    dism /online /add-package /packagepath:C:\updatefix\expanded\Windows8.1-KB4338424-x64.cab
  9. You have now fixed the issue and all updates should now be cleanly installing via Windows Updates!

Leave a comment and let me know if this worked for you!

Aug 202018
 

An all too common problem is when users report e-mail delays ranging from 5 to 15 minutes. When troubleshing these types of issues, you’ll notice this commonly occurs when receiving e-mails from organizations that use Office 365. Specifically this occurs due to greylisting.

Why does this happen

You’re organization is using greylisting on your e-mail proxy/SMTP relay to reduce spam. Greylisting temporarily rejects the first send of an e-mail and waits for the sending server to re-transmit the message. This process usually takes around 5-15 minutes to complete. Greylisting is used because spammers won’t re-transmit the message, which leads to a massive reduction of spam messages coming through.

Once the sending server retransmits, the sending server IP address is added to your firewalls “safe senders” whitelist. From this point on the IP address (or server) will not be subject to greylisting (and any subsequent e-mails).

Office 365 has hundreds, if not thousands (possibly 10’s of thousands) of servers they use to transmit e-mail. The chance of multiple e-mails being sent from a single server is very slim, therefor greylisting is applied to every IP (server) that is sending e-mail because it’s different. Each e-mail from an Office 365 user can take 5-15 minutes, since a new server is used every time.

How to resolve

You’ll need to configure and add an exception to your e-mail proxy/SMTP relay/firewall. This exception can be based off domain, DNS name of sending server, or IP address ranges.

Scroll down for instructions on how to create an exception on a Sophos UTM.

Domain Exception

If you use domain based exceptions, you’ll need to configure these manually for each sending domain that you want your firewall to skip greylist checking. This is a very manual process, which requires lots of human intervention to continuously update your greylist exception.

DNS FQDN of MX Server

This method is the easiest, however most firewall or UTM’s will now allow these types of exceptions since a number of DNS queries will be needed everytime an e-mail comes in. One DNS query on the MX record, and then another DNS query on the DNS host contained in the MX record. If you can configure this type of exception, you’ll want to configure it as below:

*-com.mail.protection.outlook.com

IP Address Range

This is the best method. To create an IP address range exception, we’ll need a copy of all the IP address ranges or IP address spaces that Office 365 uses to send mail. This list can be found at: https://docs.microsoft.com/en-us/office365/SecurityCompliance/eop/exchange-online-protection-ip-addresses.

We’ll need to create an exception that skips greylist checking on the IP addresses outlined in the above link. This will stop any greylist checking on e-mails from Office 365 servers.

In my case, I use a Sophos UTM firewall, and to create an exception I had to do the following:

  1. Log on to the Webmin interface.
  2. Select “Email Protection”, then “STMP” on the left hand side, then “Exceptions” tab at the top.

    Sophos UTM E-Mail and SMTP Exception List

    Sophos UTM E-Mail and SMTP Exception List

  3. Create a “New Exception List” and call it “Office 365 GreylistWhitelist”.
  4. Check the “Greylisting” box under “Antispam”, and then check the “For these source hosts/networks”.

    Sophos UTM SMTP Create Exception

    Sophos UTM SMTP Create Exception

  5. Click the “+” button, and call the Network Definition “Exchange365-EOP-Group”. Change the type to “Network Group”.
  6. Click the “+” button in the members section, and start adding the IP spaces. Repeat this for each IP space (in total I added 23). Each network name (IP address space) requires a unique name, I named mine “Exchange365-EOP1” through “Exchange365-EOP23”.

    Sophos UTM SMTP Configure Exception

    Sophos UTM SMTP Configure Exception

  7. Click Save on the Network Group, and click Save on the exception.
  8. Enable the Exception

    Sophos UTM SMTP Exception Rule

    Sophos UTM SMTP Exception Rule

  9. Completed! You’ve now made the exception and delays should no longer occur.
Apr 292018
 
Directory Services Restore Mode

Running Veeam Backup and Replication, a Microsoft Windows Server Domain Controller may boot in to safe mode and directory services restore mode.

About a week ago, I loaded up Veeam Backup and Replication in to my test environment. It’s a fantastic product, and it’s working great, however today I had a little bit of an issue with a DC running Windows Server 2016 Server Core.

I woke up to a notification that the backup failed due to a VSS snapshot issue. Now I know that VSS can be a little picky at times, so I decided to restart the guest VM. Upon restarting, she came back up, was pingable, and appeared to be running fine, however the backup kept failing with new errors, the event log was looking very strange on the server, and numerous services that were set to automatic were not starting up.

This specific server was installed using Server Core mode, so it has no GUI and is administered via command prompt over RDP, or via remote management utilities. Once RDP’ing in to the server, I noticed the “Safe Mode” branding on each corner of the display, this was very odd. I restarted the server again, this time manually trying to start Active Directory Services manually via services.msc.

This presented:

Event ID: 16652
Source: Directory-Services-SAM
General Description: The domain controller is booting to directory services restore mode.

Screenshot:

Directory Services Restore Mode

The domain controller is booting to directory services restore mode.

 

This surprised me (and scared me for that matter). I immediately started searching the internet to find out what would have caused this…

To my relief, I read numerous sites that advise that when an active backup is running on a guest VM which is a domain controller, Veeam activates directory services restore mode temporarily, so in the event of a restore, it will boot to this mode automatically. In my case, the switch was not changed back during the backup failure.

Running the following command in a command prompt, verifies that the safeboot switch is set to dsrepair enabled:

bcdedit /v

To disable directory services restore mode, type the following in a command prompt:

bcdedit /deletevalue safeboot

Restart the server and the issue should be resolved!