Aug 202018
 

An all too common problem is when users report e-mail delays ranging from 5 to 15 minutes. When troubleshing these types of issues, you’ll notice this commonly occurs when receiving e-mails from organizations that use Office 365. Specifically this occurs due to greylisting.

Why does this happen

You’re organization is using greylisting on your e-mail proxy/SMTP relay to reduce spam. Greylisting temporarily rejects the first send of an e-mail and waits for the sending server to re-transmit the message. This process usually takes around 5-15 minutes to complete. Greylisting is used because spammers won’t re-transmit the message, which leads to a massive reduction of spam messages coming through.

Once the sending server retransmits, the sending server IP address is added to your firewalls “safe senders” whitelist. From this point on the IP address (or server) will not be subject to greylisting (and any subsequent e-mails).

Office 365 has hundreds, if not thousands (possibly 10’s of thousands) of servers they use to transmit e-mail. The chance of multiple e-mails being sent from a single server is very slim, therefor greylisting is applied to every IP (server) that is sending e-mail because it’s different. Each e-mail from an Office 365 user can take 5-15 minutes, since a new server is used every time.

How to resolve

You’ll need to configure and add an exception to your e-mail proxy/SMTP relay/firewall. This exception can be based off domain, DNS name of sending server, or IP address ranges.

Scroll down for instructions on how to create an exception on a Sophos UTM.

Domain Exception

If you use domain based exceptions, you’ll need to configure these manually for each sending domain that you want your firewall to skip greylist checking. This is a very manual process, which requires lots of human intervention to continuously update your greylist exception.

DNS FQDN of MX Server

This method is the easiest, however most firewall or UTM’s will now allow these types of exceptions since a number of DNS queries will be needed everytime an e-mail comes in. One DNS query on the MX record, and then another DNS query on the DNS host contained in the MX record. If you can configure this type of exception, you’ll want to configure it as below:

*-com.mail.protection.outlook.com

IP Address Range

This is the best method. To create an IP address range exception, we’ll need a copy of all the IP address ranges or IP address spaces that Office 365 uses to send mail. This list can be found at: https://docs.microsoft.com/en-us/office365/SecurityCompliance/eop/exchange-online-protection-ip-addresses.

We’ll need to create an exception that skips greylist checking on the IP addresses outlined in the above link. This will stop any greylist checking on e-mails from Office 365 servers.

In my case, I use a Sophos UTM firewall, and to create an exception I had to do the following:

  1. Log on to the Webmin interface.
  2. Select “Email Protection”, then “STMP” on the left hand side, then “Exceptions” tab at the top.

    Sophos UTM E-Mail and SMTP Exception List

    Sophos UTM E-Mail and SMTP Exception List

  3. Create a “New Exception List” and call it “Office 365 GreylistWhitelist”.
  4. Check the “Greylisting” box under “Antispam”, and then check the “For these source hosts/networks”.

    Sophos UTM SMTP Create Exception

    Sophos UTM SMTP Create Exception

  5. Click the “+” button, and call the Network Definition “Exchange365-EOP-Group”. Change the type to “Network Group”.
  6. Click the “+” button in the members section, and start adding the IP spaces. Repeat this for each IP space (in total I added 23). Each network name (IP address space) requires a unique name, I named mine “Exchange365-EOP1” through “Exchange365-EOP23”.

    Sophos UTM SMTP Configure Exception

    Sophos UTM SMTP Configure Exception

  7. Click Save on the Network Group, and click Save on the exception.
  8. Enable the Exception

    Sophos UTM SMTP Exception Rule

    Sophos UTM SMTP Exception Rule

  9. Completed! You’ve now made the exception and delays should no longer occur.
Apr 292018
 
Directory Services Restore Mode

Running Veeam Backup and Replication, a Microsoft Windows Server Domain Controller may boot in to safe mode and directory services restore mode.

About a week ago, I loaded up Veeam Backup and Replication in to my test environment. It’s a fantastic product, and it’s working great, however today I had a little bit of an issue with a DC running Windows Server 2016 Server Core.

I woke up to a notification that the backup failed due to a VSS snapshot issue. Now I know that VSS can be a little picky at times, so I decided to restart the guest VM. Upon restarting, she came back up, was pingable, and appeared to be running fine, however the backup kept failing with new errors, the event log was looking very strange on the server, and numerous services that were set to automatic were not starting up.

This specific server was installed using Server Core mode, so it has no GUI and is administered via command prompt over RDP, or via remote management utilities. Once RDP’ing in to the server, I noticed the “Safe Mode” branding on each corner of the display, this was very odd. I restarted the server again, this time manually trying to start Active Directory Services manually via services.msc.

This presented:

Event ID: 16652
Source: Directory-Services-SAM
General Description: The domain controller is booting to directory services restore mode.

Screenshot:

Directory Services Restore Mode

The domain controller is booting to directory services restore mode.

 

This surprised me (and scared me for that matter). I immediately started searching the internet to find out what would have caused this…

To my relief, I read numerous sites that advise that when an active backup is running on a guest VM which is a domain controller, Veeam activates directory services restore mode temporarily, so in the event of a restore, it will boot to this mode automatically. In my case, the switch was not changed back during the backup failure.

Running the following command in a command prompt, verifies that the safeboot switch is set to dsrepair enabled:

bcdedit /v

To disable directory services restore mode, type the following in a command prompt:

bcdedit /deletevalue safeboot

Restart the server and the issue should be resolved!

Jan 212018
 
Azure AD

This weekend I configured Azure AD Connect for pass through authentication for my on-premise Active Directory domain. This was a first for me and extremely easy to do, however there was a few issues with my firewall and SSL content filtering and scanning rules which was blocking the connection. I figured I’d create a post providing some information you’ll need to get this setup and running quickly.

In my environment, I have a Sophos UTM firewall which provides firewall services (port blocking), as well as HTTP and HTTPs scanning and filtering (web filtering).

The Problem

After running the Azure AD Connect wizard, all went good however there was an error at the end of the wizard notifying that synchronization was configured but is not occurring due to firewall. It provided a link for more information (that actually didn’t really contain the information needed).

While this issue is occurring, you’ll notice:

-Azure AD Connect in the Azure portal is reporting that pass-through authentication is Enabled, however after expanding the item, the Authentication Agent reports a status of Inactive on your internal domain controllers.

-In the Event log, under “Applications and Services Logs”, then “Microsoft”, then “AzureADConnect”, then “AuthenticationAgent”, and finally “Admin”, you’ll see the following error event:

Event ID: 12019

Source: Microsoft Azure AD Connect Authentication Agent (Microsoft-AzureADConnect-AuthenticationAgent)

Event:
The Connector stopped working because the client certificate is not valid. Uninstall the Connector and install it again. Request ID: '{WAJAJAJA-OHYA-YAAA-YAAAA-WAKAKAKAKAKAKAK}'

This event log above is due to the SSL and HTTPs content filtering.

-Azure Pass-Through authentication won’t work

The Fix

After doing some research, I came up with the following list of ports and hosts you’ll need to allow unfiltered to a specific list of hosts.

Ports

The following ports are used by Azure AD Connect:

Port 443 – SSL

Port 5671 – TCP (From the host running the Azure AD Connect to Internet)

Hosts (DNS Hosts)

Here’s the host list:

*blob.core.windows.net
*servicebus.windows.net
*adhybridhealth.azure.com
*management.azure.com
*policykeyservice.dc.ad.msft.net
*login.windows.net
*login.microsoftonline.com
*secure.aadcdn.microsoftonline-p.com
*microsoftonline.com
*windows.net
*msappproxy.net
*mscrl.microsoft.com
*crl.microsoft.com
*ocsp.msocsp.com
*www.microsoft.com

If you’re running a Sophos UTM like I am, you’ll need to create an HTTP(s) scanning exception and then import this list in to a rule “Matching these URLs”:

^https?://([A-Za-z0-9.-]*\.)?blob.core.windows.net/
^https?://([A-Za-z0-9.-]*\.)?servicebus.windows.net/
^https?://([A-Za-z0-9.-]*\.)?adhybridhealth.azure.com/
^https?://([A-Za-z0-9.-]*\.)?management.azure.com/
^https?://([A-Za-z0-9.-]*\.)?policykeyservice.dc.ad.msft.net/
^https?://([A-Za-z0-9.-]*\.)?login.windows.net/
^https?://([A-Za-z0-9.-]*\.)?login.microsoftonline.com/
^https?://([A-Za-z0-9.-]*\.)?secure.aadcdn.microsoftonline-p.com/
^https?://([A-Za-z0-9.-]*\.)?microsoftonline.com/
^https?://([A-Za-z0-9.-]*\.)?windows.net/
^https?://([A-Za-z0-9.-]*\.)?msappproxy.net/
^https?://([A-Za-z0-9.-]*\.)?mscrl.microsoft.com/
^https?://([A-Za-z0-9.-]*\.)?crl.microsoft.com/
^https?://([A-Za-z0-9.-]*\.)?ocsp.msocsp.com/
^https?://([A-Za-z0-9.-]*\.)?www.microsoft.com/

The exception I created skips:

  • Authentication
  • Caching
  • Antivirus
  • Extension Blocking
  • MIME type blocking
  • URL Filter
  • Content Removal
  • SSL Scanning
  • Certificate trust check
  • Certificate date check

After creating the exceptions, I restarted the “Microsoft Azure AD Connect Authentication Agent”. The errors stopped and Azure AD Pass-through started to function correctly! Also the status of the Authentication Agent now reports a status of active.

Jan 142018
 

The Problem

In the latest updates and versions of Microsoft Office 2016, I found a bug where when a user adds a new on-premise Microsoft Exchange 2016 account, it will repeatedly prompt for a username and password and ultimately fail if you hit cancel (no matter how many times you enter credentials). This was on the internal LAN on a domain joined workstation.

I did the usual checks:

  • Check Virtualdirectory configuration on Exchange
  • Check Virtualdirectory configuration on IIS (Internet Information Services)
  • Check Autodiscover DNS entries, InternalURL and ExternalURL configuration
  • Check for SCP inside of domain

All the of the above came back fine and were configured properly.

I have numerous other Outlook 2016 clients configured and working (installed as older versions, but have been updated), so I used those to troubleshoot (same scenario, domain joined on internal LAN and external WAN). After spending 10 hours ripping apart everything, confirming configuration, I noticed that when using the “Test Email Autoconfiguration…” (holding CTRL while right clicking on Outlook tray icon), that the e-mail clients had a skewed order for checking autodiscovery.

The e-mail clients were actually trying to authenticate with Office365 before my own on-premise Exchange Server (domain SCP or autodiscover records). This is absolutely bizarre! After spending 2 hours googling (I couldn’t find anything), I finally stumbled across this document and found an interesting piece of information:

https://support.microsoft.com/en-ca/help/3211279/outlook-2016-implementation-of-autodiscover

“Outlook uses a set of heuristics to determine whether the user account provided comes from Office 365. If Outlook determines confidently that you are an O365 user, a try is made to retrieve the Autodiscover payload from the known O365 endpoints (typically https://autodiscover-s.outlook.com/autodiscover/autodiscover.xml or https://autodiscover-s.partner.outlook.cn/autodiscover/autodiscover.xml). If this step does not retrieve a payload, Outlook moves to step 5.”

WTF?!?!?

So while this doesn’t explain why this happened, it explains what’s happening. I believe this is what’s happening as my working clients are trying to Autodisocver with Office365 first…

I went ahead an created a registry value to control the policy for “ExcludeExplicitO365Endpoint“. After configuring the registry key, I noticed that Autodiscover was now functioning properly and checking SCP and autodiscover DNS records first. I have no idea why the “heuristics” determined I was an Office365 user, but I’m not (I do have access to Office365 as a partner, but don’t use it and don’t have it configured). This may effect other partners, or users that utilize some O365 services…

The Fix

To fix this issue, create a text file and copy/paste this text below.

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Microsoft\Office\16.0\Outlook\AutoDiscover]
"ExcludeExplicitO365Endpoint"=dword:00000001

Then save it, and rename it as ExcludeExplicitO365Endpoint.reg and run it (this will import the applicable registry key). ONLY DO THIS if you are using an Exchange On-Premise account, and not a Office365 or hosted exchange account.

After this, the issue was completely fixed. If you know what you’re doing, you can also use Outlook GPO settings and deploy this to a vast number of systems using Group Policy.

Nov 062017
 

Something that has bothered me for a very long time has been the fact that mobile devices (using Microsoft Exchange ActiveSync), automatically send read receipts if the sender has requested it without prompting the user. This means that if someone sends you an e-mail, requests and read receipt, and you open it on your mobile device; it will send a read receipt without prompting you or giving you a choice in the matter.

This is bad for a number of reasons such as spam (this is a big one, where they try to validate e-mail addresses), legal reasons, you don’t have the time to respond and don’t want a read receipt sent yet, or you simply don’t send read receipts…

Now, with Microsoft Exchange 2016 you can disable this so that mobile devices don’t automatically send these read receipts out. It’s a simple procedure using Outlook on the web (previously known as Outlook Web Access, a.k.a OWA).

To disable automatic read-receipts:

  1. Log on to your OWA (Outlook on the web) server.
  2. Click on settings (the gear) on the top rightOutlook on the Web (OWA) Settings
  3. Expand the “General” settings menu, and select “Mobile Devices” (as shown below)
    Outlook on the Web (OWA) Settings Pane
  4. Check the checkbox for “Don’t send read receipts for messages read on devices that use Exchange ActiveSync”.
    Don't send read receipts for messages read on devices that use Exchange ActiveSync

You’re done!

Nov 062017
 

After doing a migration from Microsoft Exchange 2013 to Exchange 2016 I noticed that my Offline Address Book (OAB) wasn’t being made available to Outlook clients.

When trying to perform a manual download (Send and Receive -> Download Address Book), it wasn’t in the list. Also when using the “Test EMail AutoConfiguration..” (by holding CTRL and right click on Outlook System Tray icon) to examine the AutoDiscover information, there was no OAB URL (OABUrl in XML) being sent to the clients.

I spent 3 hours trying to find out why this was happening (I assumed it was configuration and/or IIS authentication related). All my virtual directories and URLs were fine, and the OAB was being generated fine without any issues. It simply wasn’t being passed to Outlook clients. I couldn’t find any references of this occurring to other users.

I finally discovered that the “WebDistributionEnabled” configuration flag was marked to False, when it needs to be marked as True. This flag when set to true, allows it to be distributed (Note/Fun Fact: There’s a separate and different flag for older Exchange versions where the OAB is inside of the Public Folder Store). There’s also a different flag “GlobalWebDistributionEnabled”, which is recommended to be enabled as well on Exchange 2016. When setting this second flag to True, it also sets the first one above to True as well.

To fix it we’ll use Exchange PowerShell:

Let’s find the name of your Offline Address Book by running the command below:

Get-OfflineAddressBook

Exchange Offline Address Book Get-OfflineAddressBook

Now let’s set the “GlobalWebDistributionEnabled” flag to True using this next command:

Set-OfflineAddressBook -Identity “Default Offline Address Book (Ex2016)” -GlobalWebDistributionEnabled $true

Set Offline Address Book Microsoft Exchange 2016 Default Set-OfflineAddressBook

And finally let’s confirm to make sure the changes take effect and look for the values of “GlobalWebDistributionEnabled” and “WebDistributionEnabled” using the command:

Get-OfflineAddressBook | fl

Get-OfflineAddressBook WebDistributionEnabled GlobalDistributionEnabled

 

After making the above changes I recommend issuing an “iisreset” or restarting your Exchange Server. There will also be a delay where you’ll need to wait for your Outlook clients to refresh their autodiscover configuration. You can run the “Test Email AutoConfiguration…” to see if the OAB is now being passed to your clients.

Nov 052017
 

 

Update – January 8th 2018: After upgrading from Exchange 2016 CU7 to Exchange 2016 CU8 and restarting the server, the password prompt was occurring again on internal/external domain joined computers. Stay posted for more information.

Update – January 13th 2018: If you upgrade to any new CU versions (CU8 or higher), I would recommend resetting all your virtual directories to REVERSE the configuration advised below. On CU8, new issues arose and were resolved by fully resetting (restoring to default) the virtualdirectory configuration, and then re configuring them with the appropriate URL values. The fix below was NOT applied and is NOT needed on CU8 or later.

Update – January 14th 2018: If you still receive password prompts, you Outlook 2016 client may be trying to autoconfigure with Office365 instead of your on-premise Exchange deployment. This is due to the autodiscover order being skewed on a new Outlook 2016 update. Please see https://www.stephenwagner.com/2018/01/14/cannot-create-exchange-2016-account-office-2016-due-repeated-password-prompts/ for more information and a fix for this.

 

Original Article:

Today I came across an issue that I experienced with Microsoft Exchange 2013, and Microsoft Exchange 2016. The issue relates to using MAPI over HTTP with Microsoft Outlook 2016 (however I’m sure this affects earlier versions) clients.

MAPI over HTTP is used standard on Exchange 2016, or can be enabled manually on Exchange 2013 via running the command:

Set-OrganizationConfig -MapiHttpEnabled $true

 

You’ll notice that when domain joined computers are internal to the LAN, they will work fine and there will not be any password prompts coming from Microsoft Outlook. However, when a domain joined user leaves the LAN and is external to the network, they will start to receive password prompts like below:

Outlook Password Prompt

 

After spending hours, I found this fix resolves the situation and applies to both Exchange 2013, and Exchange 2016:

 

Open up Exchange PowerShell and change the authentication methods on the MAPI virtual directory. We will be removing the negotiate authentication mechanism. Use the command below:

Set-MapiVirtualDirectory -Identity “YOURSERVERNAME\mapi (Default Web Site)” -ExternalURL https://YOURSERVERNAME.YOURDOMAIN.com/mapi -IISAuthenticationMethods NTLM,OAuth

We now need to modify the Authentication settings inside of IIS to remove Negotiate from both the mapi and EWS directories. The command above may have removed it from mapi, but it’s still good to confirm and we still need to change it for EWS. Open IIS Manager, Expand “Default Web Site”. Select “EWS” on the left hand side, and then select “Authentication” on the Right side as shown below:

IIS Manager Left Pane

Select Windows Authentication and then click “Providers” on the right Action Pane. Now remove “Neogiate” from the list so that only NTLM remains, as shown below:

IIS Manager Authentication Providers

Repeat for the mapi on the left as well (Select “Default Web Site”, select “mapi” on the left hand side, and then select “Authentication” on the right side), and confirm that only NTLM is in the list of providers.

Open up command prompt and type “IISRESET” to reload IIS, or restart your Exchange Server!

Nov 052017
 

Around the end of September, I posted a blog article talking about Outlook 2016 prompting for password credentials due to a Office 2016 click to run update bug when using Microsoft Exchange. While they did just recently fix this by deploying a new update, I have since come across a new bug in the latest update.

I noticed multiple computers with Outlook 2016 Version 1710 (Build 8625.2121 Click-to-Run) started getting stuck with the Outlook icon on the system tray showing that it was sending and receiving. When opening Outlook, and hitting Send and Receive, nothing is shown.

When you hold down CTRL and right click on the Outlook icon, choose “Connection Status…”, then select the “Local Mailbox” tab, you’ll notice it gets stuck on “Email@address.com – Saving synchronization log” (as seen below).

I went ahead and tried all the usual troubleshooting steps like deleting and recreating the OST and Outlook Mail Profiles, but it still had no effect. I went ahead and completely uninstalled Microsoft Office, and reinstalled an older version. The issue DID NOT occur on the older version. Once updating to the latest, the bug re-occurred.

I’ve been scouring the internet for 2 days now trying to find information on this however I haven’t received any. This is most likely a new bug produced in the update that resolved the last bug. I will be posting updates when I hear more.

UPDATE November 7th, 2017 (Thanks Tony):

Microsoft has acknowledged that an MVP has reported this issue to the team. They are investigating.

Oct 182017
 

After installing Windows 10 Fall Creators Update (Windows 10 Version 1709), I’m noticing that on one of my multi-monitor machines it’s showing blue colors as purple on one of the displays.

This is very visible when highlighting text, viewing the blue Facebook logo and banner, or any other blue content. When dragging something across both displays (window is shown on both displays) you can see the color differences. However, one interesting thing, is that when dragging from one display to the other, for the last 10% or so when moving, it’ll quickly change to the proper blue before leaving the display, which means this is software related since it will briefly show the proper blue.

After spending over an hour troubleshooting, it’s totally unrelated to monitor drivers (color configurations), video drivers, etc… and I cannot find any configuration to fix this. Also, searching on the internet I cannot find any other occurrences.

Please comment if you have any information, or are experiencing the same issue!

 

Update: I’ve seen 2 other posts of people reporting issues with colors, but no one is going in to detail. I’ve found that the color differences actually show up in screenshots as well (the color changes depending on which display it’s on).

 

Update October 25th, 2017 – Very odd update… I went ahead and tried using the “Calibrate display color”, and while I didn’t really change any settings, on completion of the wizard the colors are now fixed on my display. I’m thinking this is an issue or bug in Windows 10 Fall Creators Update.

Oct 182017
 

Well, it’s October 18th 2017 and the Fall Creators update (Feature update to Windows 10, version 1709) is now available for download. In my particular environment, I use WSUS to deploy and manage updates.

Update: It’s now May 2018, and this article also applies to Windows 10 April 2018 update version 1803 as well!

I went ahead earlier today and approved the updates for deployment, however I noticed an issue on multiple Windows 10 machines, where the Windows Update client would get stuck on Downloading updates 0% status.

I checked a bunch of things, but noticed that it simply couldn’t download the updates from my WSUS server. Further investigation found that the feature updates are packaged in .esd files and IIS may not be able to serve these properly without a minor modification. I remember applying this fix in the past, however I’m assuming it was removed by a prior update on my Windows Server 2012 R2 server.

If you are experiencing this issue, here’s the fix:

  1. On your server running WSUS and IIS, open up the IIS manager.
  2. Expand Sites, and select “WSUS Administration”
  3. On the right side, under IIS, select “MIME Types”
  4. Make sure there is not a MIME type for .esd, if there is, you’re having a different issue, if not, continue with the instructions.
  5. Click on “Add” on the right Actions pane.
  6. File name extension will be “.esd” (without quotations), and MIME type will be “application/octet-stream” (without quotations).
  7. Reset IIS or restart WSUS/IIS server

You’ll notice the clients will now update without a problem! Happy Updating!