Jan 182018
 

The Problem

I run a Sophos UTM firewall appliance in my VMware vSphere environment and noticed the other day that I was getting warnings on the space used on the ESXi host for the thin-provisioned vmdk file for the guest VM. I thought “Hey, this is weird”, so I enabled SSH and logged in to check my volumes. Everything looked fine and my disk usage was great! So what gives?

After spending some more time troubleshooting and not finding much, I thought to myself “What if it’s not unmapping unused blocks from the vmdk to the host ESXi machine?”. What is unampping you ask? When files get deleted in a guest VM, the free blocks aren’t automatically “unmapped” and released back to the host hypervisor in some cases.

Two things need to happen:

  1. The guest VM has to release these blocks (notify the hypervisor that it’s not using them, making the vmdk smaller)
  2. The host has to reclaim these and issue the unmap command to the storage (freeing up the space on the SAN/storage itself)

On a side note: In ESXi 6.5 and when using VMFS version 6 (VMFS6), the datastores can be configured for automatic unmapping. You can still kick it off manually, but many administrators would prefer it to happen automatically in the background with low priority (low I/O).

Most of my guest VMs automatically do the first step (releasing the blocks back to the host). On Windows this occurs with the defrag utility which issues trim commands and “trims” the volumes. On linux this occurs with the fstrim command. All my guest VMs do this automatically with the exception being the Sophos UTM appliance.

The fix

First, a warning: Enable SSH on the Sophos UTM at your own risk. You need to know what you are doing, this also may pose a security risk and should be disabled once your are finished. You’ll need to “su” to root once you log in with the “loginuser” account.

This fix not only applies to the Sophos UTM, but most other linux based guest virtual machines.

Now to fix the issue, I used the “df” command which provides a list of the filesystems, their mount points, and storage free for those fileystems. I’ve included an example below (this wasn’t the full list):

hostname:/root # df
Filesystem                       1K-blocks     Used Available Use% Mounted on
/dev/sda6                          5412452  2832960   2281512  56% /
udev                               3059712       72   3059640   1% /dev
tmpfs                              3059712      100   3059612   1% /dev/shm
/dev/sda1                           338875    15755    301104   5% /boot
/dev/sda5                         98447760 13659464  79513880  15% /var/storage
/dev/sda7                        129002700  4624468 117474220   4% /var/log
/dev/sda8                          5284460   274992   4717988   6% /tmp
/dev                               3059712       72   3059640   1% /var/storage/chroot-clientlessvpn/dev


You’ll need to run the fstrim command on every mountpoint for file systems “/dev/sdaX” (X means you’ll be doing this for multiple mountpoints). In the example above, you’ll need to run it on “/”, “/boot”, “/var/storage”, “/var/log”, “/tmp”, and any other mountpoints that use “/dev/sdaX” filesytems.

Two examples:

fstrim / -v

fstrim / -v

 

 

fstrim /var/storage -v

fstrim /var/storage -v

 

 

Again, you’ll repeat this for all mount points for your /dev/sdaX storage (X is replaced with the volume number). The command above only works with mountpoints, and not the actual device mappings.

Time to release the unused blocks to the SAN:

The above completes the first step of releasing the storage back to the host. Now you can either let the automatic unmap occur slowly overtime if you’re using VMFS6, or you can manually kick it off. I decided to manually kick it off using the steps I have listed at: https://www.stephenwagner.com/2017/02/07/vmfs-unmap-command-on-vsphere-6-5-with-vmfs-6-runs-repeatedly/

You’ll need to use esxcli to do this. I simply enabled SSH on my ESXi hosts temporarily.

Please note: Using the unmap command on ESXi hosts is very storage I/O intensive. Do this during maintenance window, or at a time of low I/O as this will perform MAJOR I/O on your hosts…

I issue the command (replace “DATASTORENAME” with the name of your datastore):

esxcli storage vmfs unmap --volume-label=DATASTORENAME --reclaim-unit=8

This could run for hours, possibly days depending on your “reclaim-unit” size (this is the block size of the unit you’re trying to reclaim from the VMFS file-system). In this example I choose 8, but most people do something larger like 100, or 200 to reduce the load and time for the command to complete (lower values look for smaller chunks of free space, so the command takes longer to execute).

I let this run for 2 hours on a 10TB datastore, however it may take way longer (possibly 6+ hours, to days).

Finally, not only are we are left with a smaller vmdk file, but we’ve released the space back to the SAN as well!

Jan 142018
 

The Problem

In the latest updates and versions of Microsoft Office 2016, I found a bug where when a user adds a new on-premise Microsoft Exchange 2016 account, it will repeatedly prompt for a username and password and ultimately fail if you hit cancel (no matter how many times you enter credentials). This was on the internal LAN on a domain joined workstation.

I did the usual checks:

  • Check Virtualdirectory configuration on Exchange
  • Check Virtualdirectory configuration on IIS (Internet Information Services)
  • Check Autodiscover DNS entries, InternalURL and ExternalURL configuration
  • Check for SCP inside of domain

All the of the above came back fine and were configured properly.

I have numerous other Outlook 2016 clients configured and working (installed as older versions, but have been updated), so I used those to troubleshoot (same scenario, domain joined on internal LAN and external WAN). After spending 10 hours ripping apart everything, confirming configuration, I noticed that when using the “Test Email Autoconfiguration…” (holding CTRL while right clicking on Outlook tray icon), that the e-mail clients had a skewed order for checking autodiscovery.

The e-mail clients were actually trying to authenticate with Office365 before my own on-premise Exchange Server (domain SCP or autodiscover records). This is absolutely bizarre! After spending 2 hours googling (I couldn’t find anything), I finally stumbled across this document and found an interesting piece of information:

https://support.microsoft.com/en-ca/help/3211279/outlook-2016-implementation-of-autodiscover

“Outlook uses a set of heuristics to determine whether the user account provided comes from Office 365. If Outlook determines confidently that you are an O365 user, a try is made to retrieve the Autodiscover payload from the known O365 endpoints (typically https://autodiscover-s.outlook.com/autodiscover/autodiscover.xml or https://autodiscover-s.partner.outlook.cn/autodiscover/autodiscover.xml). If this step does not retrieve a payload, Outlook moves to step 5.”

WTF?!?!?

So while this doesn’t explain why this happened, it explains what’s happening. I believe this is what’s happening as my working clients are trying to Autodisocver with Office365 first…

I went ahead an created a registry value to control the policy for “ExcludeExplicitO365Endpoint“. After configuring the registry key, I noticed that Autodiscover was now functioning properly and checking SCP and autodiscover DNS records first. I have no idea why the “heuristics” determined I was an Office365 user, but I’m not (I do have access to Office365 as a partner, but don’t use it and don’t have it configured). This may effect other partners, or users that utilize some O365 services…

The Fix

To fix this issue, create a text file and copy/paste this text below.

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Microsoft\Office\16.0\Outlook\AutoDiscover]
"ExcludeExplicitO365Endpoint"=dword:00000001

Then save it, and rename it as ExcludeExplicitO365Endpoint.reg and run it (this will import the applicable registry key). ONLY DO THIS if you are using an Exchange On-Premise account, and not a Office365 or hosted exchange account.

After this, the issue was completely fixed. If you know what you’re doing, you can also use Outlook GPO settings and deploy this to a vast number of systems using Group Policy.

Jan 092018
 
HPe iLo Registered to Remote Support Insight Online

Many months ago, I configured the HPe Insight Online – Direct Connect on all my HPe Proliant DL360p Gen8 servers running VMware vSphere 6.5. This service is available with active support contracts (warranties), and allows your servers to “phone home” to HPe for free. This allows service and health information to be broadcast to your HPe passport and support account, to pro-actively manage, monitor, and maintain your servers. Information on the service can be found at https://www.hpe.com/ca/en/services/remote-it-support.html.

This is all pretty cool, but does it work? Read below!

I woke up this morning to notifications from my own monitoring system that a fan failure had occurred on one of my HPe Proliant server ESXi hosts. All my servers have fan redundancy so the server continued to run without problems. Scrolling through my other overnight e-mails, I also see e-mails from HPe acknowledging a support case that I had created. I had long since forgot that I configured Insight Online direct connect, so it actually took a few minutes for me to put two and two together. The server by itself took care of everything!

After reviewing all these e-mails, logging in to the HPe support portal, I had realized that the server by itself had:

  1. Identified a fan failure
  2. Sent diagnostic data off to HPe support
  3. Created an HPe support ticket and case
  4. HPe support engineers looked up the serial and part number of the server, and assigned a replacement part for the fan to be dispatched to me

I called in to HPe support, mentioned this was the first time this had ever happened and asked if there was anything additional I needed to provide. All the engineer asked, was whether I wanted an engineer to replace the part, or if I was comfortable replacing the part myself (of course I want to replace it myself). That was it!

This is VERY interesting and cool technology. I can see this being extremely valuable for customers who have 4 hour response contracts with their HPe equipment.

I’ve provided some screenshots below to show the process.

HPe Case Management E-Mail

HPe iLo Registered to Remote Support Insight Online

eRS Active Health Report Sent

HPe Remote Support Direct Connect Service Event

HPe Insight Online Automated Case

Jan 062018
 

Last night I updated my VMware VDI envionrment to VMware Horizon 7.4.0. For the most part the upgrade went smooth, however I discovered an issue (probably unrelated to the upgrade itself, and more so just previously overlooked). When connecting with Google Chrome to  VMware Horizon HTML Access via the UAG (Unified Access Gateway), an error pops up after pressing the button saying “Failed to connected to the connection server”.

The Problem:

This error pops up ONLY when using Chrome, and ONLY when connecting through the UAG. If you use a different browser (Firefox, IE), this issue will not occur. If you connect using Chrome to the connection server itself, this issue will not occur. It took me hours to find out what was causing this as virtually nothing popped up when searching for a solution.

Finally I stumbled across a VMware document that mentions on View Connection Server instances and security servers that reside behind a gateway (such as a UAG, or Access Point), the instance must be aware of the address in which browsers will connect to the gateway for HTML access.

The VMware document is here: https://docs.vmware.com/en/VMware-Horizon-7/7.0/com.vmware.horizon-view.installation.doc/GUID-FE26A9DE-E344-42EC-A1EE-E1389299B793.html

To resolve this:

On the view connection server, create a file called “locked.properties” in “install_directory\VMware\VMware View\Server\sslgateway\conf\”.

If you have a single UAG/Access Point, populate this file with:

portalHost=view-gateway.example.com

If you have multiple UAG/Access Points, populate the file with:

portalHost.1=view-gateway-1.example.com
portalHost.2=view-gateway-2.example.com

Restart the server

The issue should now be resolved!

On a side note, I also deleted my VMware Unified Access Gateways VMs and deployed the updated version that ship with Horizon 7.4.0. This means I deployed VMware Unified Access Gateway 3.2.0. There was an issue importing the configuration from the export backup I took from the previous version, so I had to configure from scratch (installing certificates, configuring URLs, etc…), be aware of this issue importing configuration.

 

Dec 152017
 

The Challenge

Finding a cost-effective SIP trunk provider in Canada can be one of the biggest challenges that a business may have when trying to adopt VoIP technology. This is also a common problem for VoIP PBX re-sellers, as it’s hard to find a good provider to refer.

Back in 2007, just a year in to running my own company, my telecommunication and voice requirements massively grew. I needed a phone system to handle multiple extensions, call forwarding, conference rooms, follow-me services, rings groups, and needed the ability for staff and contractors to have their phones (and extensions) in remote offices or home offices. Also, I was travelling quite frequently so I needed to be able to have an extension running on my smartphone (so it would appear as if I was at the office, and to save on international roaming and long distance costs).

Implementing a VoIP PBX phone system handled all of this, and was very easy to implement however finding an SIP trunk provider was not. Originally I was using FXO/FXS adapters to pipe analog lines in to my PBX, however I wasn’t happy with the quality or the complexity of a solution. I wanted a true 100% digital, and 100% Canadian hosted solution.

The Solution (The Review)

After spending months researching providers, I came across a company called Iristel. There were numerous great reviews on the internet, and most importantly they had a following of Trixbox (Asterisk) users, so I could verify they would work with my PBX. They were a Canadian company (important to me), who provided SIP trunks at a great cost. I signed up for service, and tech support was actually amazing at providing assistance for configuring the SIP trunks with my Asterisk PBX, their sales staff was pretty awesome as well!

Here’s where the review gets boring (which is a good thing), I’ve been using them for around 10 years now, and everything has always just worked! I think in 10 years, I may have experience a single 1-2 hours of downtime, and this was due to a compatibility issue with Asterisk and their SIP gateways caused by an update (SIP registration bug). In this one-off case, tech support was immediately available and made configuration changes to resolve this issue. Outstanding service to say the least!

Over the years, I’ve also re-configured and deployed new PBXs. I’m now using FreePBX, and Iristel is still working great! AND YES, Iristel supports T38 faxing!

I would definitely recommend Iristel as a your VoIP SIP provider for your business digital telephony needs!

 

Feel free to reach out (comment) if you have any questions about my review, or the quality of the services.

Nov 062017
 

Something that has bothered me for a very long time has been the fact that mobile devices (using Microsoft Exchange ActiveSync), automatically send read receipts if the sender has requested it without prompting the user. This means that if someone sends you an e-mail, requests and read receipt, and you open it on your mobile device; it will send a read receipt without prompting you or giving you a choice in the matter.

This is bad for a number of reasons such as spam (this is a big one, where they try to validate e-mail addresses), legal reasons, you don’t have the time to respond and don’t want a read receipt sent yet, or you simply don’t send read receipts…

Now, with Microsoft Exchange 2016 you can disable this so that mobile devices don’t automatically send these read receipts out. It’s a simple procedure using Outlook on the web (previously known as Outlook Web Access, a.k.a OWA).

To disable automatic read-receipts:

  1. Log on to your OWA (Outlook on the web) server.
  2. Click on settings (the gear) on the top rightOutlook on the Web (OWA) Settings
  3. Expand the “General” settings menu, and select “Mobile Devices” (as shown below)
    Outlook on the Web (OWA) Settings Pane
  4. Check the checkbox for “Don’t send read receipts for messages read on devices that use Exchange ActiveSync”.
    Don't send read receipts for messages read on devices that use Exchange ActiveSync

You’re done!

Nov 062017
 

After doing a migration from Microsoft Exchange 2013 to Exchange 2016 I noticed that my Offline Address Book (OAB) wasn’t being made available to Outlook clients.

When trying to perform a manual download (Send and Receive -> Download Address Book), it wasn’t in the list. Also when using the “Test EMail AutoConfiguration..” (by holding CTRL and right click on Outlook System Tray icon) to examine the AutoDiscover information, there was no OAB URL (OABUrl in XML) being sent to the clients.

I spent 3 hours trying to find out why this was happening (I assumed it was configuration and/or IIS authentication related). All my virtual directories and URLs were fine, and the OAB was being generated fine without any issues. It simply wasn’t being passed to Outlook clients. I couldn’t find any references of this occurring to other users.

I finally discovered that the “WebDistributionEnabled” configuration flag was marked to False, when it needs to be marked as True. This flag when set to true, allows it to be distributed (Note/Fun Fact: There’s a separate and different flag for older Exchange versions where the OAB is inside of the Public Folder Store). There’s also a different flag “GlobalWebDistributionEnabled”, which is recommended to be enabled as well on Exchange 2016. When setting this second flag to True, it also sets the first one above to True as well.

To fix it we’ll use Exchange PowerShell:

Let’s find the name of your Offline Address Book by running the command below:

Get-OfflineAddressBook

Exchange Offline Address Book Get-OfflineAddressBook

Now let’s set the “GlobalWebDistributionEnabled” flag to True using this next command:

Set-OfflineAddressBook -Identity “Default Offline Address Book (Ex2016)” -GlobalWebDistributionEnabled $true

Set Offline Address Book Microsoft Exchange 2016 Default Set-OfflineAddressBook

And finally let’s confirm to make sure the changes take effect and look for the values of “GlobalWebDistributionEnabled” and “WebDistributionEnabled” using the command:

Get-OfflineAddressBook | fl

Get-OfflineAddressBook WebDistributionEnabled GlobalDistributionEnabled

 

After making the above changes I recommend issuing an “iisreset” or restarting your Exchange Server. There will also be a delay where you’ll need to wait for your Outlook clients to refresh their autodiscover configuration. You can run the “Test Email AutoConfiguration…” to see if the OAB is now being passed to your clients.

Nov 052017
 

 

Update – January 8th 2018: After upgrading from Exchange 2016 CU7 to Exchange 2016 CU8 and restarting the server, the password prompt was occurring again on internal/external domain joined computers. Stay posted for more information.

Update – January 13th 2018: If you upgrade to any new CU versions (CU8 or higher), I would recommend resetting all your virtual directories to REVERSE the configuration advised below. On CU8, new issues arose and were resolved by fully resetting (restoring to default) the virtualdirectory configuration, and then re configuring them with the appropriate URL values. The fix below was NOT applied and is NOT needed on CU8 or later.

Update – January 14th 2018: If you still receive password prompts, you Outlook 2016 client may be trying to autoconfigure with Office365 instead of your on-premise Exchange deployment. This is due to the autodiscover order being skewed on a new Outlook 2016 update. Please see https://www.stephenwagner.com/2018/01/14/cannot-create-exchange-2016-account-office-2016-due-repeated-password-prompts/ for more information and a fix for this.

 

Original Article:

Today I came across an issue that I experienced with Microsoft Exchange 2013, and Microsoft Exchange 2016. The issue relates to using MAPI over HTTP with Microsoft Outlook 2016 (however I’m sure this affects earlier versions) clients.

MAPI over HTTP is used standard on Exchange 2016, or can be enabled manually on Exchange 2013 via running the command:

Set-OrganizationConfig -MapiHttpEnabled $true

 

You’ll notice that when domain joined computers are internal to the LAN, they will work fine and there will not be any password prompts coming from Microsoft Outlook. However, when a domain joined user leaves the LAN and is external to the network, they will start to receive password prompts like below:

Outlook Password Prompt

 

After spending hours, I found this fix resolves the situation and applies to both Exchange 2013, and Exchange 2016:

 

Open up Exchange PowerShell and change the authentication methods on the MAPI virtual directory. We will be removing the negotiate authentication mechanism. Use the command below:

Set-MapiVirtualDirectory -Identity “YOURSERVERNAME\mapi (Default Web Site)” -ExternalURL https://YOURSERVERNAME.YOURDOMAIN.com/mapi -IISAuthenticationMethods NTLM,OAuth

We now need to modify the Authentication settings inside of IIS to remove Negotiate from both the mapi and EWS directories. The command above may have removed it from mapi, but it’s still good to confirm and we still need to change it for EWS. Open IIS Manager, Expand “Default Web Site”. Select “EWS” on the left hand side, and then select “Authentication” on the Right side as shown below:

IIS Manager Left Pane

Select Windows Authentication and then click “Providers” on the right Action Pane. Now remove “Neogiate” from the list so that only NTLM remains, as shown below:

IIS Manager Authentication Providers

Repeat for the mapi on the left as well (Select “Default Web Site”, select “mapi” on the left hand side, and then select “Authentication” on the right side), and confirm that only NTLM is in the list of providers.

Open up command prompt and type “IISRESET” to reload IIS, or restart your Exchange Server!

Nov 052017
 

Around the end of September, I posted a blog article talking about Outlook 2016 prompting for password credentials due to a Office 2016 click to run update bug when using Microsoft Exchange. While they did just recently fix this by deploying a new update, I have since come across a new bug in the latest update.

I noticed multiple computers with Outlook 2016 Version 1710 (Build 8625.2121 Click-to-Run) started getting stuck with the Outlook icon on the system tray showing that it was sending and receiving. When opening Outlook, and hitting Send and Receive, nothing is shown.

When you hold down CTRL and right click on the Outlook icon, choose “Connection Status…”, then select the “Local Mailbox” tab, you’ll notice it gets stuck on “Email@address.com – Saving synchronization log” (as seen below).

I went ahead and tried all the usual troubleshooting steps like deleting and recreating the OST and Outlook Mail Profiles, but it still had no effect. I went ahead and completely uninstalled Microsoft Office, and reinstalled an older version. The issue DID NOT occur on the older version. Once updating to the latest, the bug re-occurred.

I’ve been scouring the internet for 2 days now trying to find information on this however I haven’t received any. This is most likely a new bug produced in the update that resolved the last bug. I will be posting updates when I hear more.

UPDATE November 7th, 2017 (Thanks Tony):

Microsoft has acknowledged that an MVP has reported this issue to the team. They are investigating.

Oct 272017
 

I went to re-deploy some vDP appliances today and noticed a newer version was made available a few months ago (vSphere Data Protection 6.1.5). After downloading the vSphereDataProtection-6.1.5.ova file, I went to deploy it to my vSphere cluster and it failed due to an invalid certificate and a message reading “The OVF package is signed with an invalid certificate”.

I went ahead and downloaded the certificate to see what was wrong with it. While the publisher was valid, the certificate was only valid from September 5th, 2016 to September 7th, 2017, and today was October 27th, 2017. It looks like the guys at VMware should have generated a new cert before releasing it.

 

 

To resolve this, you need to repackage the OVA file and skip the certificate using the VMware Open Virtualization Format Tool (ovftool) available at https://code.vmware.com/tool/ovf/4.1.0

Once you download and install this, the executable can be found in your Program Files\VMware\VMware OVF Tool folder.

Open a command prompt and change to the above directory and run the following:

ovftool.exe –skipManifestCheck c:\folder\vSphereDataProtection-6.1.5.ova c:\folder\vdpgood.ova

This command will repackage and remove the certificate from the OVA and save it as the new file named vdpgood.ova above. Afterwards deploy it to your vSphere environment and all should be working!