Apr 292018
 
Directory Services Restore Mode

Running Veeam Backup and Replication, a Microsoft Windows Server Domain Controller may boot in to safe mode and directory services restore mode.

About a week ago, I loaded up Veeam Backup and Replication in to my test environment. It’s a fantastic product, and it’s working great, however today I had a little bit of an issue with a DC running Windows Server 2016 Server Core.

I woke up to a notification that the backup failed due to a VSS snapshot issue. Now I know that VSS can be a little picky at times, so I decided to restart the guest VM. Upon restarting, she came back up, was pingable, and appeared to be running fine, however the backup kept failing with new errors, the event log was looking very strange on the server, and numerous services that were set to automatic were not starting up.

This specific server was installed using Server Core mode, so it has no GUI and is administered via command prompt over RDP, or via remote management utilities. Once RDP’ing in to the server, I noticed the “Safe Mode” branding on each corner of the display, this was very odd. I restarted the server again, this time manually trying to start Active Directory Services manually via services.msc.

This presented:

Event ID: 16652
Source: Directory-Services-SAM
General Description: The domain controller is booting to directory services restore mode.

Screenshot:

Directory Services Restore Mode

The domain controller is booting to directory services restore mode.

 

This surprised me (and scared me for that matter). I immediately started searching the internet to find out what would have caused this…

To my relief, I read numerous sites that advise that when an active backup is running on a guest VM which is a domain controller, Veeam activates directory services restore mode temporarily, so in the event of a restore, it will boot to this mode automatically. In my case, the switch was not changed back during the backup failure.

Running the following command in a command prompt, verifies that the safeboot switch is set to dsrepair enabled:

bcdedit /v

To disable directory services restore mode, type the following in a command prompt:

bcdedit /deletevalue safeboot

Restart the server and the issue should be resolved!

Jan 212018
 
Azure AD

This weekend I configured Azure AD Connect for pass through authentication for my on-premise Active Directory domain. This was a first for me and extremely easy to do, however there was a few issues with my firewall and SSL content filtering and scanning rules which was blocking the connection. I figured I’d create a post providing some information you’ll need to get this setup and running quickly.

In my environment, I have a Sophos UTM firewall which provides firewall services (port blocking), as well as HTTP and HTTPs scanning and filtering (web filtering).

The Problem

After running the Azure AD Connect wizard, all went good however there was an error at the end of the wizard notifying that synchronization was configured but is not occurring due to firewall. It provided a link for more information (that actually didn’t really contain the information needed).

While this issue is occurring, you’ll notice:

-Azure AD Connect in the Azure portal is reporting that pass-through authentication is Enabled, however after expanding the item, the Authentication Agent reports a status of Inactive on your internal domain controllers.

-In the Event log, under “Applications and Services Logs”, then “Microsoft”, then “AzureADConnect”, then “AuthenticationAgent”, and finally “Admin”, you’ll see the following error event:

Event ID: 12019

Source: Microsoft Azure AD Connect Authentication Agent (Microsoft-AzureADConnect-AuthenticationAgent)

Event:
The Connector stopped working because the client certificate is not valid. Uninstall the Connector and install it again. Request ID: '{WAJAJAJA-OHYA-YAAA-YAAAA-WAKAKAKAKAKAKAK}'

This event log above is due to the SSL and HTTPs content filtering.

-Azure Pass-Through authentication won’t work

The Fix

After doing some research, I came up with the following list of ports and hosts you’ll need to allow unfiltered to a specific list of hosts.

Ports

The following ports are used by Azure AD Connect:

Port 443 – SSL

Port 5671 – TCP (From the host running the Azure AD Connect to Internet)

Hosts (DNS Hosts)

Here’s the host list:

*blob.core.windows.net
*servicebus.windows.net
*adhybridhealth.azure.com
*management.azure.com
*policykeyservice.dc.ad.msft.net
*login.windows.net
*login.microsoftonline.com
*secure.aadcdn.microsoftonline-p.com
*microsoftonline.com
*windows.net
*msappproxy.net
*mscrl.microsoft.com
*crl.microsoft.com
*ocsp.msocsp.com
*www.microsoft.com

If you’re running a Sophos UTM like I am, you’ll need to create an HTTP(s) scanning exception and then import this list in to a rule “Matching these URLs”:

^https?://([A-Za-z0-9.-]*\.)?blob.core.windows.net/
^https?://([A-Za-z0-9.-]*\.)?servicebus.windows.net/
^https?://([A-Za-z0-9.-]*\.)?adhybridhealth.azure.com/
^https?://([A-Za-z0-9.-]*\.)?management.azure.com/
^https?://([A-Za-z0-9.-]*\.)?policykeyservice.dc.ad.msft.net/
^https?://([A-Za-z0-9.-]*\.)?login.windows.net/
^https?://([A-Za-z0-9.-]*\.)?login.microsoftonline.com/
^https?://([A-Za-z0-9.-]*\.)?secure.aadcdn.microsoftonline-p.com/
^https?://([A-Za-z0-9.-]*\.)?microsoftonline.com/
^https?://([A-Za-z0-9.-]*\.)?windows.net/
^https?://([A-Za-z0-9.-]*\.)?msappproxy.net/
^https?://([A-Za-z0-9.-]*\.)?mscrl.microsoft.com/
^https?://([A-Za-z0-9.-]*\.)?crl.microsoft.com/
^https?://([A-Za-z0-9.-]*\.)?ocsp.msocsp.com/
^https?://([A-Za-z0-9.-]*\.)?www.microsoft.com/

The exception I created skips:

  • Authentication
  • Caching
  • Antivirus
  • Extension Blocking
  • MIME type blocking
  • URL Filter
  • Content Removal
  • SSL Scanning
  • Certificate trust check
  • Certificate date check

After creating the exceptions, I restarted the “Microsoft Azure AD Connect Authentication Agent”. The errors stopped and Azure AD Pass-through started to function correctly! Also the status of the Authentication Agent now reports a status of active.

Jan 142018
 

The Problem

In the latest updates and versions of Microsoft Office 2016, I found a bug where when a user adds a new on-premise Microsoft Exchange 2016 account, it will repeatedly prompt for a username and password and ultimately fail if you hit cancel (no matter how many times you enter credentials). This was on the internal LAN on a domain joined workstation.

I did the usual checks:

  • Check Virtualdirectory configuration on Exchange
  • Check Virtualdirectory configuration on IIS (Internet Information Services)
  • Check Autodiscover DNS entries, InternalURL and ExternalURL configuration
  • Check for SCP inside of domain

All the of the above came back fine and were configured properly.

I have numerous other Outlook 2016 clients configured and working (installed as older versions, but have been updated), so I used those to troubleshoot (same scenario, domain joined on internal LAN and external WAN). After spending 10 hours ripping apart everything, confirming configuration, I noticed that when using the “Test Email Autoconfiguration…” (holding CTRL while right clicking on Outlook tray icon), that the e-mail clients had a skewed order for checking autodiscovery.

The e-mail clients were actually trying to authenticate with Office365 before my own on-premise Exchange Server (domain SCP or autodiscover records). This is absolutely bizarre! After spending 2 hours googling (I couldn’t find anything), I finally stumbled across this document and found an interesting piece of information:

https://support.microsoft.com/en-ca/help/3211279/outlook-2016-implementation-of-autodiscover

“Outlook uses a set of heuristics to determine whether the user account provided comes from Office 365. If Outlook determines confidently that you are an O365 user, a try is made to retrieve the Autodiscover payload from the known O365 endpoints (typically https://autodiscover-s.outlook.com/autodiscover/autodiscover.xml or https://autodiscover-s.partner.outlook.cn/autodiscover/autodiscover.xml). If this step does not retrieve a payload, Outlook moves to step 5.”

WTF?!?!?

So while this doesn’t explain why this happened, it explains what’s happening. I believe this is what’s happening as my working clients are trying to Autodisocver with Office365 first…

I went ahead an created a registry value to control the policy for “ExcludeExplicitO365Endpoint“. After configuring the registry key, I noticed that Autodiscover was now functioning properly and checking SCP and autodiscover DNS records first. I have no idea why the “heuristics” determined I was an Office365 user, but I’m not (I do have access to Office365 as a partner, but don’t use it and don’t have it configured). This may effect other partners, or users that utilize some O365 services…

The Fix

To fix this issue, create a text file and copy/paste this text below.

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Microsoft\Office\16.0\Outlook\AutoDiscover]
"ExcludeExplicitO365Endpoint"=dword:00000001

Then save it, and rename it as ExcludeExplicitO365Endpoint.reg and run it (this will import the applicable registry key). ONLY DO THIS if you are using an Exchange On-Premise account, and not a Office365 or hosted exchange account.

Keep in mind that autodiscover also queries the domain root (domain.com), before querying the autodiscover host (autodiscover.domain.com). If you want to stop both the Office365 autodiscover and the root domain autodiscover challenge, use the following below:

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Microsoft\Office\16.0\Outlook\AutoDiscover]
"ExcludeExplicitO365Endpoint"=dword:00000001
"ExcludeHttpsRootDomain"=dword:00000001

You’ll notice that we also set “ExcludeHttpsRootDomain” to “1” which stops it from checking the root domain.

After this, the issue was completely fixed. If you know what you’re doing, you can also use Outlook GPO settings and deploy this to a vast number of systems using Group Policy.

Additional Note (added November 2nd, 2018)

While reading numerous documents covering autodiscovery, I also came across an article that went in to detail with particulars as to how Mapi over HTTP functions. Even with the above, when accessing Outlook externall from the domain, you may still notice a single password prompt for the first time you log in externally.

After reading through documentation, I found that this is most likely because the first user account login (the very first time the user logged in on the computer), the username format of “DOMAIN\Username” was used, and not the UPN. The documentation mentioned that this may fail the negotiation, which will require a single password prompt on autodiscovery. This issue can be avoided by using the users UPN ([email protected]) to log in for the first time on the system.

Please note that the UPN must match the user’s e-mail address.

Nov 062017
 

Something that has bothered me for a very long time has been the fact that mobile devices (using Microsoft Exchange ActiveSync), automatically send read receipts if the sender has requested it without prompting the user. This means that if someone sends you an e-mail, requests and read receipt, and you open it on your mobile device; it will send a read receipt without prompting you or giving you a choice in the matter.

This is bad for a number of reasons such as spam (this is a big one, where they try to validate e-mail addresses), legal reasons, you don’t have the time to respond and don’t want a read receipt sent yet, or you simply don’t send read receipts…

Now, with Microsoft Exchange 2016 you can disable this so that mobile devices don’t automatically send these read receipts out. It’s a simple procedure using Outlook on the web (previously known as Outlook Web Access, a.k.a OWA).

To disable automatic read-receipts:

  1. Log on to your OWA (Outlook on the web) server.
  2. Click on settings (the gear) on the top rightOutlook on the Web (OWA) Settings
  3. Expand the “General” settings menu, and select “Mobile Devices” (as shown below)
    Outlook on the Web (OWA) Settings Pane
  4. Check the checkbox for “Don’t send read receipts for messages read on devices that use Exchange ActiveSync”.
    Don't send read receipts for messages read on devices that use Exchange ActiveSync

You’re done!

Nov 062017
 

After doing a migration from Microsoft Exchange 2013 to Exchange 2016 I noticed that my Offline Address Book (OAB) wasn’t being made available to Outlook clients.

When trying to perform a manual download (Send and Receive -> Download Address Book), it wasn’t in the list. Also when using the “Test EMail AutoConfiguration..” (by holding CTRL and right click on Outlook System Tray icon) to examine the AutoDiscover information, there was no OAB URL (OABUrl in XML) being sent to the clients.

I spent 3 hours trying to find out why this was happening (I assumed it was configuration and/or IIS authentication related). All my virtual directories and URLs were fine, and the OAB was being generated fine without any issues. It simply wasn’t being passed to Outlook clients. I couldn’t find any references of this occurring to other users.

I finally discovered that the “WebDistributionEnabled” configuration flag was marked to False, when it needs to be marked as True. This flag when set to true, allows it to be distributed (Note/Fun Fact: There’s a separate and different flag for older Exchange versions where the OAB is inside of the Public Folder Store). There’s also a different flag “GlobalWebDistributionEnabled”, which is recommended to be enabled as well on Exchange 2016. When setting this second flag to True, it also sets the first one above to True as well.

To fix it we’ll use Exchange PowerShell:

Let’s find the name of your Offline Address Book by running the command below:

Get-OfflineAddressBook

Exchange Offline Address Book Get-OfflineAddressBook

Now let’s set the “GlobalWebDistributionEnabled” flag to True using this next command:

Set-OfflineAddressBook -Identity “Default Offline Address Book (Ex2016)” -GlobalWebDistributionEnabled $true

Set Offline Address Book Microsoft Exchange 2016 Default Set-OfflineAddressBook

And finally let’s confirm to make sure the changes take effect and look for the values of “GlobalWebDistributionEnabled” and “WebDistributionEnabled” using the command:

Get-OfflineAddressBook | fl

Get-OfflineAddressBook WebDistributionEnabled GlobalDistributionEnabled

 

After making the above changes I recommend issuing an “iisreset” or restarting your Exchange Server. There will also be a delay where you’ll need to wait for your Outlook clients to refresh their autodiscover configuration. You can run the “Test Email AutoConfiguration…” to see if the OAB is now being passed to your clients.

Nov 052017
 

 

Update – January 8th 2018: After upgrading from Exchange 2016 CU7 to Exchange 2016 CU8 and restarting the server, the password prompt was occurring again on internal/external domain joined computers. Stay posted for more information.

Update – January 13th 2018: If you upgrade to any new CU versions (CU8 or higher), I would recommend resetting all your virtual directories to REVERSE the configuration advised below. On CU8, new issues arose and were resolved by fully resetting (restoring to default) the virtualdirectory configuration, and then re configuring them with the appropriate URL values. The fix below was NOT applied and is NOT needed on CU8 or later.

Update – January 14th 2018: If you still receive password prompts, you Outlook 2016 client may be trying to autoconfigure with Office365 instead of your on-premise Exchange deployment. This is due to the autodiscover order being skewed on a new Outlook 2016 update. Please see https://www.stephenwagner.com/2018/01/14/cannot-create-exchange-2016-account-office-2016-due-repeated-password-prompts/ for more information and a fix for this.

 

Original Article:

Today I came across an issue that I experienced with Microsoft Exchange 2013, and Microsoft Exchange 2016. The issue relates to using MAPI over HTTP with Microsoft Outlook 2016 (however I’m sure this affects earlier versions) clients.

MAPI over HTTP is used standard on Exchange 2016, or can be enabled manually on Exchange 2013 via running the command:

Set-OrganizationConfig -MapiHttpEnabled $true

 

You’ll notice that when domain joined computers are internal to the LAN, they will work fine and there will not be any password prompts coming from Microsoft Outlook. However, when a domain joined user leaves the LAN and is external to the network, they will start to receive password prompts like below:

Outlook Password Prompt

 

After spending hours, I found this fix resolves the situation and applies to both Exchange 2013, and Exchange 2016:

 

Open up Exchange PowerShell and change the authentication methods on the MAPI virtual directory. We will be removing the negotiate authentication mechanism. Use the command below:

Set-MapiVirtualDirectory -Identity “YOURSERVERNAME\mapi (Default Web Site)” -ExternalURL https://YOURSERVERNAME.YOURDOMAIN.com/mapi -IISAuthenticationMethods NTLM,OAuth

We now need to modify the Authentication settings inside of IIS to remove Negotiate from both the mapi and EWS directories. The command above may have removed it from mapi, but it’s still good to confirm and we still need to change it for EWS. Open IIS Manager, Expand “Default Web Site”. Select “EWS” on the left hand side, and then select “Authentication” on the Right side as shown below:

IIS Manager Left Pane

Select Windows Authentication and then click “Providers” on the right Action Pane. Now remove “Neogiate” from the list so that only NTLM remains, as shown below:

IIS Manager Authentication Providers

Repeat for the mapi on the left as well (Select “Default Web Site”, select “mapi” on the left hand side, and then select “Authentication” on the right side), and confirm that only NTLM is in the list of providers.

Open up command prompt and type “IISRESET” to reload IIS, or restart your Exchange Server!

Nov 052017
 

Around the end of September, I posted a blog article talking about Outlook 2016 prompting for password credentials due to a Office 2016 click to run update bug when using Microsoft Exchange. While they did just recently fix this by deploying a new update, I have since come across a new bug in the latest update.

I noticed multiple computers with Outlook 2016 Version 1710 (Build 8625.2121 Click-to-Run) started getting stuck with the Outlook icon on the system tray showing that it was sending and receiving. When opening Outlook, and hitting Send and Receive, nothing is shown.

When you hold down CTRL and right click on the Outlook icon, choose “Connection Status…”, then select the “Local Mailbox” tab, you’ll notice it gets stuck on “[email protected] – Saving synchronization log” (as seen below).

I went ahead and tried all the usual troubleshooting steps like deleting and recreating the OST and Outlook Mail Profiles, but it still had no effect. I went ahead and completely uninstalled Microsoft Office, and reinstalled an older version. The issue DID NOT occur on the older version. Once updating to the latest, the bug re-occurred.

I’ve been scouring the internet for 2 days now trying to find information on this however I haven’t received any. This is most likely a new bug produced in the update that resolved the last bug. I will be posting updates when I hear more.

UPDATE November 7th, 2017 (Thanks Tony):

Microsoft has acknowledged that an MVP has reported this issue to the team. They are investigating.

Oct 182017
 

After installing Windows 10 Fall Creators Update (Windows 10 Version 1709), I’m noticing that on one of my multi-monitor machines it’s showing blue colors as purple on one of the displays.

This is very visible when highlighting text, viewing the blue Facebook logo and banner, or any other blue content. When dragging something across both displays (window is shown on both displays) you can see the color differences. However, one interesting thing, is that when dragging from one display to the other, for the last 10% or so when moving, it’ll quickly change to the proper blue before leaving the display, which means this is software related since it will briefly show the proper blue.

After spending over an hour troubleshooting, it’s totally unrelated to monitor drivers (color configurations), video drivers, etc… and I cannot find any configuration to fix this. Also, searching on the internet I cannot find any other occurrences.

Please comment if you have any information, or are experiencing the same issue!

 

Update: I’ve seen 2 other posts of people reporting issues with colors, but no one is going in to detail. I’ve found that the color differences actually show up in screenshots as well (the color changes depending on which display it’s on).

 

Update October 25th, 2017 – Very odd update… I went ahead and tried using the “Calibrate display color”, and while I didn’t really change any settings, on completion of the wizard the colors are now fixed on my display. I’m thinking this is an issue or bug in Windows 10 Fall Creators Update.

Oct 182017
 

Well, it’s October 18th 2017 and the Fall Creators update (Feature update to Windows 10, version 1709) is now available for download. In my particular environment, I use WSUS to deploy and manage updates.

Update: It’s now May 2018, and this article also applies to Windows 10 April 2018 update version 1803 as well!

Update: It’s now October 2018, and this article also applies to Windows 10 October 2018 update version 1809 as well!

Update: It’s now May 2019, and this article also applies to Windows 10 May 2019 update version 1903 as well!

I went ahead earlier today and approved the updates for deployment, however I noticed an issue on multiple Windows 10 machines, where the Windows Update client would get stuck on Downloading updates 0% status.

I checked a bunch of things, but noticed that it simply couldn’t download the updates from my WSUS server. Further investigation found that the feature updates are packaged in .esd files and IIS may not be able to serve these properly without a minor modification. I remember applying this fix in the past, however I’m assuming it was removed by a prior update on my Windows Server 2012 R2 server.

If you are experiencing this issue, here’s the fix:

  1. On your server running WSUS and IIS, open up the IIS manager.
  2. Expand Sites, and select “WSUS Administration”
  3. On the right side, under IIS, select “MIME Types”
  4. Make sure there is not a MIME type for .esd, if there is, you’re having a different issue, if not, continue with the instructions.
  5. Click on “Add” on the right Actions pane.
  6. File name extension will be “.esd” (without quotations), and MIME type will be “application/octet-stream” (without quotations).
  7. Reset IIS or restart WSUS/IIS server

You’ll notice the clients will now update without a problem! Happy Updating!

Sep 292017
 

There is a new issue starting to be visible in the last couple days that I’ve noticed across 3 fully patched systems (Windows 10 running Outlook 2016 connecting to Exchange 2013).

When using Microsoft Outlook 2016 with Microsoft Exchange 2013, a password prompt becomes visible when opening an attachment in an e-mail. The attachment will open, however the prompt occurs after it’s opened, and only appears if an attachment is opened in the first place. The prompt will not appear if an attachment is never opened or highlighted (selected).

Outlook Password Prompt

When entering AD credentials, the prompt keeps re-appearing. When you hit cancel, Outlook will continue to function. You may also see the prompt shown below.

Exchange Password Prompt

After troubleshooting, I can confirm this is NOT related to any of the traditional “Outlook password prompt” issues that users normally experience due to misconfiguration, and I have a feeling this is related to either an Outlook 2016 update, or an update for Microsoft Windows 10 (and/or Microsoft Windows 7).

I’ve only found one other mention of this occurring on the internet which appeared a day ago, where multiple users are experience the same issue with Microsoft Office 365 with Microsoft Outlook 2016 with multiple operating systems (Windows 10 and Windows 7).

Microsoft Office Version: 1708 (Build 8431.2079)

As of right now I have no information on a fix, but I wanted to post this before other admins start ripping apart their Exchange servers trying to resolve this.

Please see below for a fix!

Update October 2nd, 2017: I’ve read that someone used the downgrade guide from Microsoft and downgraded their Outlook 2016 client to an earlier “Click-to-Run” 2016 version. This stopped the password prompt so it appears this issue has to do with the latest updates for Microsoft Office (Office 2016 and Office 365).

Update October 23rd, 2017: Still not fix, however Microsoft has finally acknowledged this issue. Information on their workaround can be found here. Essentially they’re recommending downgrading to a previous “Click to Run” version of Office.

Update November 3rd, 2017: Our Reader AC reported that Microsoft released a statement saying that they addressed this issue in the most recent flights (updates revisions for a line of products). I updated my Office 2016 Click-to-Run instance, and I am no longer receiving the password prompts. I will update in a few hours to confirm it stays this way!

To Update:
1) Open an Office Product (Such as word, outlook, etc…)
2) Click File
3) Click “Office Account”
4) Click “Update Options” on the right side
5) Click “Update Now” from the drop down

Update November 5th, 2017: I can confirm that the latest updates have fully resolved this issue, but create a new issue as well.