Nov 062017
 

Something that has bothered me for a very long time has been the fact that mobile devices (using Microsoft Exchange ActiveSync), automatically send read receipts if the sender has requested it without prompting the user. This means that if someone sends you an e-mail, requests and read receipt, and you open it on your mobile device; it will send a read receipt without prompting you or giving you a choice in the matter.

This is bad for a number of reasons such as spam (this is a big one, where they try to validate e-mail addresses), legal reasons, you don’t have the time to respond and don’t want a read receipt sent yet, or you simply don’t send read receipts…

Now, with Microsoft Exchange 2016 you can disable this so that mobile devices don’t automatically send these read receipts out. It’s a simple procure using Outlook on the web (previously known as Outlook Web Access, a.k.a OWA).

To disable automatic read-receipts:

  1. Log on to your OWA (Outlook on the web) server.
  2. Click on settings (the gear) on the top right
  3. Expand the “General” settings menu, and select “Mobile Devices” (as shown below)
  4. Check the checkbox for “Don’t send read receipts for messages read on devices that use Exchange ActiveSync”.

You’re done!

Nov 062017
 

After doing a migration from Microsoft Exchange 2013 to Exchange 2016 I noticed that my Offline Address Book (OAB) wasn’t being made available to Outlook clients.

When trying to perform a manual download (Send and Receive -> Download Address Book), it wasn’t in the list. Also when using the “Test EMail AutoConfiguration..” (by holding CTRL and right click on Outlook System Tray icon) to examine the AutoDiscover information, there was no OAB URL (OABUrl in XML) being sent to the clients.

I spent 3 hours trying to find out why this was happening (I assumed it was configuration and/or IIS authentication related). All my virtual directories and URLs were fine, and the OAB was being generated fine without any issues. It simply wasn’t being passed to Outlook clients. I couldn’t find any references of this occurring to other users.

I finally discovered that the “WebDistributionEnabled” configuration flag was marked to False, when it needs to be marked as True. This flag when set to true, allows it to be distributed (Note/Fun Fact: There’s a separate and different flag for older Exchange versions where the OAB is inside of the Public Folder Store). There’s also a different flag “GlobalWebDistributionEnabled”, which is recommended to be enabled as well on Exchange 2016. When setting this second flag to True, it also sets the first one above to True as well.

To fix it we’ll use Exchange PowerShell:

Let’s find the name of your Offline Address Book by running the command below:

Get-OfflineAddressBook

Now let’s set the “GlobalWebDistributionEnabled” flag to True using this next command:

Set-OfflineAddressBook -Identity “Default Offline Address Book (Ex2016)” -GlobalWebDistributionEnabled $true

And finally let’s confirm to make sure the changes take effect and look for the values of “GlobalWebDistributionEnabled” and “WebDistributionEnabled” using the command:

Get-OfflineAddressBook | fl

 

After making the above changes I recommend issuing an “iisreset” or restarting your Exchange Server. There will also be a delay where you’ll need to wait for your Outlook clients to refresh their autodiscover configuration. You can run the “Test Email AutoConfiguration…” to see if the OAB is now being passed to your clients.

Nov 052017
 

Today I came across an issue that I experienced with Microsoft Exchange 2013, and Microsoft Exchange 2016. The issue relates to using MAPI over HTTP with Microsoft Outlook 2016 (however I’m sure this affects earlier versions) clients.

MAPI over HTTP is used standard on Exchange 2016, or can be enabled manually on Exchange 2013 via running the command:

Set-OrganizationConfig -MapiHttpEnabled $true

 

You’ll notice that when domain joined computers are internal to the LAN, they will work fine and there will not be any password prompts coming from Microsoft Outlook. However, when a domain joined user leaves the LAN and is external to the network, they will start to receive password prompts like below:

Outlook Password Prompt

 

After spending hours, I found this fix resolves the situation and applies to both Exchange 2013, and Exchange 2016:

 

Open up Exchange PowerShell and change the authentication methods on the MAPI virtual directory. We will be removing the negotiate authentication mechanism. Use the command below:

Set-MapiVirtualDirectory -Identity “YOURSERVERNAME\mapi (Default Web Site)” -ExternalURL https://YOURSERVERNAME.YOURDOMAIN.com/mapi -IISAuthenticationMethods NTLM,OAuth

We now need to modify the Authentication settings inside of IIS to remove Negotiate from both the mapi and EWS directories. The command above may have removed it from mapi, but it’s still good to confirm and we still need to change it for EWS. Open IIS Manager, Expand “Default Web Site”. Select “EWS” on the left hand side, and then select “Authentication” on the Right side as shown below:

Select Windows Authentication and then click “Providers” on the right Action Pane. Now remove “Neogiate” from the list so that only NTLM remains, as shown below:

Repeat for the mapi on the left as well (Select “Default Web Site”, select “mapi” on the left hand side, and then select “Authentication” on the right side), and confirm that only NTLM is in the list of providers.

Open up command prompt and type “IISRESET” to reload IIS, or restart your Exchange Server!

Nov 052017
 

Around the end of September, I posted a blog article talking about Outlook 2016 prompting for password credentials due to a Office 2016 click to run update bug when using Microsoft Exchange. While they did just recently fix this by deploying a new update, I have since come across a new bug in the latest update.

I noticed multiple computers with Outlook 2016 Version 1710 (Build 8625.2121 Click-to-Run) started getting stuck with the Outlook icon on the system tray showing that it was sending and receiving. When opening Outlook, and hitting Send and Receive, nothing is shown.

When you hold down CTRL and right click on the Outlook icon, choose “Connection Status…”, then select the “Local Mailbox” tab, you’ll notice it gets stuck on “Email@address.com – Saving synchronization log” (as seen below).

I went ahead and tried all the usual troubleshooting steps like deleting and recreating the OST and Outlook Mail Profiles, but it still had no effect. I went ahead and completely uninstalled Microsoft Office, and reinstalled an older version. The issue DID NOT occur on the older version. Once updating to the latest, the bug re-occurred.

I’ve been scouring the internet for 2 days now trying to find information on this however I haven’t received any. This is most likely a new bug produced in the update that resolved the last bug. I will be posting updates when I hear more.

Oct 182017
 

After installing Windows 10 Fall Creators Update (Windows 10 Version 1709), I’m noticing that on one of my multi-monitor machines it’s showing blue colors as purple on one of the displays.

This is very visible when highlighting text, viewing the blue Facebook logo and banner, or any other blue content. When dragging something across both displays (window is shown on both displays) you can see the color differences. However, one interesting thing, is that when dragging from one display to the other, for the last 10% or so when moving, it’ll quickly change to the proper blue before leaving the display, which means this is software related since it will briefly show the proper blue.

After spending over an hour troubleshooting, it’s totally unrelated to monitor drivers (color configurations), video drivers, etc… and I cannot find any configuration to fix this. Also, searching on the internet I cannot find any other occurrences.

Please comment if you have any information, or are experiencing the same issue!

 

Update: I’ve seen 2 other posts of people reporting issues with colors, but no one is going in to detail. I’ve found that the color differences actually show up in screenshots as well (the color changes depending on which display it’s on).

 

Update October 25th, 2017 – Very odd update… I went ahead and tried using the “Calibrate display color”, and while I didn’t really change any settings, on completion of the wizard the colors are now fixed on my display. I’m thinking this is an issue or bug in Windows 10 Fall Creators Update.

Oct 182017
 

Well, it’s October 18th 2017 and the Fall Creators update (Feature update to Windows 10, version 1709) is now available for download. In my particular environment, I use WSUS to deploy and manage updates.

I went ahead earlier today and approved the updates for deployment, however I noticed an issue on multiple Windows 10 machines, where the Windows Update client would get stuck on Downloading updates 0% status.

I checked a bunch of things, but noticed that it simply couldn’t download the updates from my WSUS server. Further investigation found that the feature updates are packaged in .esd files and IIS may not be able to serve these properly without a minor modification. I remember applying this fix in the past, however I’m assuming it was removed by a prior update on my Windows Server 2012 R2 server.

If you are experiencing this issue, here’s the fix:

  1. On your server running WSUS and IIS, open up the IIS manager.
  2. Expand Sites, and select “WSUS Administration”
  3. On the right side, under IIS, select “MIME Types”
  4. Make sure there is not a MIME type for .esd, if there is, you’re having a different issue, if not, continue with the instructions.
  5. Click on “Add” on the right Actions pane.
  6. File name extension will be “.esd” (without quotations), and MIME type will be “application/octet-stream” (without quotations).
  7. Reset IIS or restart WSUS/IIS server

You’ll notice the clients will not update without a problem! Happy Updating!

Sep 292017
 

There is a new issue starting to be visible in the last couple days that I’ve noticed across 3 fully patched systems (Windows 10 running Outlook 2016 connecting to Exchange 2013).

When using Microsoft Outlook 2016 with Microsoft Exchange 2013, a password prompt becomes visible when opening an attachment in an e-mail. The attachment will open, however the prompt occurs after it’s opened, and only appears if an attachment is opened in the first place. The prompt will not appear if an attachment is never opened or highlighted (selected).

Outlook Password Prompt

When entering AD credentials, the prompt keeps re-appearing. When you hit cancel, Outlook will continue to function. You may also see the prompt shown below.

Exchange Password Prompt

After troubleshooting, I can confirm this is NOT related to any of the traditional “Outlook password prompt” issues that users normally experience due to misconfiguration, and I have a feeling this is related to either an Outlook 2016 update, or an update for Microsoft Windows 10 (and/or Microsoft Windows 7).

I’ve only found one other mention of this occurring on the internet which appeared a day ago, where multiple users are experience the same issue with Microsoft Office 365 with Microsoft Outlook 2016 with multiple operating systems (Windows 10 and Windows 7).

Microsoft Office Version: 1708 (Build 8431.2079)

As of right now I have no information on a fix, but I wanted to post this before other admins start ripping apart their Exchange servers trying to resolve this.

Please see below for a fix!

Update October 2nd, 2017: I’ve read that someone used the downgrade guide from Microsoft and downgraded their Outlook 2016 client to an earlier “Click-to-Run” 2016 version. This stopped the password prompt so it appears this issue has to do with the latest updates for Microsoft Office (Office 2016 and Office 365).

Update October 23rd, 2017: Still not fix, however Microsoft has finally acknowledged this issue. Information on their workaround can be found here. Essentially they’re recommending downgrading to a previous “Click to Run” version of Office.

Update November 3rd, 2017: Our Reader AC reported that Microsoft released a statement saying that they addressed this issue in the most recent flights (updates revisions for a line of products). I updated my Office 2016 Click-to-Run instance, and I am no longer receiving the password prompts. I will update in a few hours to confirm it stays this way!

To Update:
1) Open an Office Product (Such as word, outlook, etc…)
2) Click File
3) Click “Office Account”
4) Click “Update Options” on the right side
5) Click “Update Now” from the drop down

Update November 5th, 2017: I can confirm that the latest updates have fully resolved this issue, but create a new issue as well.

Feb 182017
 

This is an issue that effects quite a few people and numerous forum threads can be found on the internet by those searching for the solution.

This can occur both when taking manual snapshots of virtual machines when one chooses “Quiesce guest filesystem”, or when using snapshot based backup applications such as vSphere Data Protection (vSphere vDP).

 

For the last couple days, one of my test VMs (Windows Server 2012 R2) has been experiencing this issue and the snapshot has been failing with the following errors:

An error occurred while taking a snapshot: Failed to quiesce the virtual machine.
An error occurred while saving the snapshot: Failed to quiesce the virtual machine.

As always with standard troubleshooting, I restarted the VM, checked for VSS provider errors, and insured that the Windows Services involved with snapshots were in their correct state and configuration. Unfortunately this had no effect, and everything was configured the way it should be.

I also tried to re-install VMWare tools, which had no effect.

PLEASE NOTE: If you experience this issue, you should confirm the services are in their correct state and configuration, as outlined in VMware KB: 1007696. Source: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007696

 

The Surprise Fix:

In the days leading up to the failure when things were running properly, I did notice that the quiesced snapshots for that VM were taking a long time process, but were still functioning correctly before the failure.

This morning during troubleshooting, I went ahead and deleted all the Windows Volume Shadow Copies which are internal and inside of the Virtual Machine itself. These are the shadow copies that the Windows guest operating system takes on it’s own filesystem (completely unrelated to VMware).

To my surprise after doing this, not only was I able to create a quiesced snapshot, but the snapshot processed almost instantly (200x faster than previously when it was functioning).

I’m assuming this was causing a high load for the VMware snapshot to process and a timeout was being hit on snapshot creation which caused the issue. While Windows volume shadow copies are unrelated to VMware snapshots, they both utilize the same VSS (Volume Shadow Copy Service) system inside of windows to function and process. One must also keep in mind that the Windows volume shadow copies will of course be part of a VMware snapshot.

PLEASE NOTE: Deleting your Windows Volume Shadow copies will delete your Windows volume snapshots inside of the virtual machine. You will lose the ability to restore files and folders from previous volume shadow copy snapshots. Be aware of what this means and what you are doing before attempting this fix.

Sep 232016
 

There’s quite a few of us that started off deploying Small Business Server (SBS2008, SBS2011) environments back in the day, loving the handy all-in-one package taking care of everything from Active Directory and Exchange, to disaster recovery and business continuity. However, some of these old environments are starting to catch up with us. I wanted to open a discussion on a big issue I had a couple years ago in one of my first migrations from SBS 2008, to Windows Server 2012 R2 with the Essentials Experience role installed, with Exchange Server 2013.

As most of you know, SBS comes packaged to push “.local” domains on initial domain configuration. This used to be considered best practice, and most of us even configured .local’s on non-SBS environments. This has never really posed any problems for us I.T. guys, except for a few configuration considerations when setting up Outlook clients, DNS, etc…

Now if you’re like me, another thing I always configured, was user accounts that didn’t match e-mail addresses. An example would be “John Doe”, with the username of “JohnD”, and the e-mail address of “John@contoso.com”. Also, our buddy John Doe would have a AD UPN JohnD@internaldomain.local (this was automatically populated on user setup)

User’s Name: John Doe

SAM Account Name: INTERNALDOMAIN\JohnD

Username: JohnD

AD UPN: JohnD@internaldomain.local

E-mail Address: John@contoso.com

 

I always liked this as it provided some protection if the users password ever got compromised (in a phishing attack, fake e-mail logon page, etc…), as the password could not actually authenticate when using the e-mail address as a username (the username was never actually provided in the attack, only e-mail).

Now let’s flash forward to this migration from SBS 2008, to Windows Server 2012 R2 with Essentials Experience, and throw Exchange 2013 in to the mix. Right off the bat, everything is working fine, Outlook 2010 is working great, Outlook 2013 is working great. Then BAM, Outlook 2016 comes out!

Outlook 2016 does not allow manual or custom configuration of Exchange accounts. They do this for “reliability” and ease of configuration. This means that you HAVE to have autodiscover setup, and working fluidly. No more manual configuration. Internally inside of the LAN this is all automatic if you configured Exchange properly, but you will have to configure autodiscover externally.

Internally on the LAN, Outlook 2016 clients have absolutely no issues, and authentication is working fine (no password prompts). However, when configuring external users, while you can eventually get it configured, the user is constantly prompted for credentials on every Outlook start.

On these password prompts, you’ll notice it’s authenticating for the users e-mail address. In this example, it’s asking for “John@contoso.com” and you enter: “INTERNALDOMAIN\JohnD” and their password, it work for the session, but keeps prompting on every fresh Outlook start.

I did massive amounts of research and seriously I could not come across one article that actually provided all the information I needed, it almost seemed as if this problem was specific to this single environment. Of course, this makes me think I have something configured incorrectly, and I literally spend forever searching for information, checking my VirtualDirectories on my Exchange server, checking logs, wasting tons and tons of time.

Finally after checking my configurations 6-10 times each and spending weeks, I realized it had nothing to do with anything configured incorrectly.

Outlook 2016 does all the configuration automatically, and expects to find everything it needs via auto discover. Putting it simple, the user’s UPN must match their e-mail address.

This means we have to change John Doe’s Active Directory UPN to match his e-mail address. The SAMAccountName still remains the same, so his login to his computer will not change, however after the change he will now be able to log in both with INTERNALDOMAIN\JohnD and John@contoso.com.

First we have to add the UPN suffix (which is the actual e-mail address domain name) to the Active Directory Domain and Trusts. Instructions are available here: https://support.microsoft.com/en-us/kb/243629

After adding your e-mail domain to the UPN suffix list. When you go in to “Active Directory Users and Computers”, and view a user’s properties, you’ll notice in the UPN section, you can drop it down and change it from internaldomain.local, to contoso.com (using my example domains). You can also change the username inside of the UPN.

 

Essentially for Johny boy, his AD properties window now looks like:

User Logon Name:

John@contoso.com (we changed the name, and chose the external domain in the drop down to the right)

User logon name (pre-Windows 2000):

INTERNALDOMAIN\ JohnD (we left this the way it was)

 

John can now login either using “INTERNALDOMAIN\JohnD” or “John@contoso.com”. As far as John is concerned we haven’t changed anything and he still logs in using the same format he always has, totally unaware of any changes.

Surprise surprise, autodiscover is now fully functioning for this user. Not only for easy configuration on mobile devices (iPhones, Windows Phones, etc…), but he can now load up Outlook 2016 away from the LAN on the Internet, type in his e-mail address, password, and BAM he’s good to go!

I am a little bit unsettled in the fact that the e-mail address now becomes a fully accepted username on the domain (for security reasons), but I guess we’re stuck with that!

 

In short, our problem is:

  1. Username doesn’t match e-mail (JohnD username, John@contoso.com email)
  2. Running Outlook 2016 and forced to use auto-discover, repeated password prompts
  3. Running .local domain internally, while using different domain externally

In Short, to fix this:

  1. Add UPN Suffix to Active Directory
  2. Change users properties so that UPN matches e-mail address, DO NOT CHANGE the old DOMAIN\Username setting

Other Considerations:

  1. Password prompts on Outlook clients can mean a whole bunch of different problems totally unrelated to this configuration and issue. Always fully diagnose the issue and confirm the issue before applying fixes. Password prompts can mean authentication problems, problems with Exchange’s virtualdirectories, issues with autodiscover, issues with certificate configuration, etc…
  2. If this is your specific issue, you can write a script to run through and update the UPNs on all the accounts. I generally don’t like scripts touching user accounts, so I’m slowly rolling out these changes per user when upgrading them to Outlook 2016. Doing this one by one as we upgrade, allows us to make sure that none of their mobile devices are affected by the UPN change.
  3. Since we are changing UPNs, this could have a major effect on any 3rd party applications that integrate with Active Directory that use UPNs. Always test, and make sure you don’t break any integration points to your 3rd party applications or line of business systems.

 

Sep 232016
 

Well, recently one of the servers I monitor and maintain in a remote oil town recently started throwing out a Windows event log warning:

Event ID: 129

Source: HpCISSs2

Description: Reset to device, \Device\RaidPort0, was issued.

 

The server is an HP ML350p Gen8 (Windows Server 2008 R2) running latest firmware and management software. It has 2 RAID Arrays (RAID1, and RAID5), and a total of 6 disks.

Researching this error, I read that most people had this occur when running the latest HP WBEM providers, as well as anti-virus software. In our case, I actually tried to downgrade to an older version, but noticed the warning still occurs. While we do have anti-virus, it’s not actively scanning (only weekly scheduled scans).

In the process of troubleshooting, I noticed that under the HP Systems Management Homepage, one of the drives in the RAID1 array, had the following stats:

Hard Read Erros:  150
Recovery Read Errors:  7
Total Seeks:  0
Seek Errors:  0

I found these numbers to be very high in my experience. None of the other drives had anything close to this (in 4 years of running, only one other disk had a read error (a single one), this disk however had tons. For some reason the drive is still reporting as operational, when I’d expect it to be marked as a predicted failure, or failed.

While all online documentation was pointing towards at locks on the array by software, from my own experience I think it was actually the array waiting for a read operation on the array, and it was this single disk that was causing a threshold to be hit in the driver, that caused a retry to recover the read operation.

Called up HPe support, I mentioned I’d like to have the drive replaced. The support engineer consulted her senior engineer and reviewed the evidence I presented (along with ADU reports, and Active Monitoring health reports), the senior engineer concurred that the drive should be replaced.

Replacing the drive resolved the issue. I’m also noticing a performance increase on the array as well.

Make sure to always check the stats on the individual components of your RAID arrays, even if everything is operating sound.