Nov 172018
 

When running VMware vSphere 6 (and later) and ESXi on your hosts with VMFS6, you may notice that auto unmap (space reclamation) is not working even though it is enabled. In addition, you’ll find that manual unmap functions still work.

Why

This is because your storage array (SAN) may have a larger unmap granularity block size than 1MB. VMFS version 6 (source) requires an unmap granularity of 1MB, and does not support automatic unmap on arrays that are larger.

For example, on the HPe MSA 2040 the page size when using virtual storage is 4MB, hence auto unmap is not supported and does not work. You can still perform automatic unmap functions on arrays with block/page sizes larger than 1MB.

Additional Information and Resources

Perform manual VMFS unmap on vSphere 6.5 and 6.7 with VMFS 6 – https://www.stephenwagner.com/2017/02/07/vmfs-unmap-command-vsphere-6-5-with-vmfs-6-auto/

vSphere 6.5, 6.7 and VMFS 6 – Change storage reclaim priority from low to medium or high – https://www.stephenwagner.com/2017/02/08/vsphere-vmfs-6-change-storage-reclaim-priority-low-medium-high/

Release unused space on host and guest filesystems with thin-provisioned Sophos UTM appliance (SW) – https://www.stephenwagner.com/2018/01/18/release-unused-space-vmdk-thin-provisioned-sophos-utm/

Nov 042018
 

This weekend I came across a big issue with my HPe MSA 2040 where one of the SAN controllers became unresponsive, and appeared to had failed because it would not boot.

It all started when I decided to clean the MSA SAN. I try to clean the components once or twice a year to remove dust and make sure it’s not getting all jammed up. Sometimes I’ll shut the entire unit down and remove the individual components, other times I’ll remove them while operating. Because of the redundancies and since I have two controllers, I can remove and clean each controller individually at separate times.

Please Note: When dusting equipment with fans, never allow the fans to spin up with compressed air as this can generate current which can damage components. Never allow compressed air flow to spin up fans.

After cleaning out the power supply’s, it came time to clean the controllers.

The Problem

As always, I logged in to the SMU to shutdown controller A (storage). I shut it down, the blue LED illuminated it was safe for removal. I then proceeded to remove it, clean it, and re-insert it. The controller came back online, and ownership of the applicable disk groups were successfully moved back. Controller A was now completed successfully. I continued to do the same for controller B: I logged in to shutdown controller B (storage). It shut down just like controller A, the blue LED removable light illuminated. I was able to remove it, clean it, and re-insert it.

However, controller B did not come back online.

After inserting controller B, the status light was flashing (as if it was booting). I waited 20 minutes with no change. The SMU on controller B was responding to HTTPS requests, however you could not log on due to the error “system is initializing”. SSH was functioning and you could log in and issue commands, however any command to get information would return “Please wait while this information is pulled from the MC controller”, and ultimately fail. The SMU on controller A would report a controller fault on controller B, and not provide any other information (including port status on controller B).

I then tried to re-seat the controller with the array still running. Gave it plenty of time with no effect.

I then removed the failed controller, shutdown the unit, powered it back on (only with controller A), and re-inserted Controller B. Again, no effect.

The Fix

At this point I’m thinking the controller may have failed or died during the cleaning process. I was just about to call HPe support for a replacement until I noticed the “Power LED” light inside of the failed controller would flash every 5 seconds while removed.

This made me start to wonder if there was an issue writing the cache to the compact flash card, or if the controller was still running off battery power but had completely frozen.

I tried these 3 things on the failed controller while it was unplugged and removed:

  1. I left the controller untouched for 1 hour out of the array (to maybe let it finish whatever it was doing while on battery power)
  2. There’s an unlabeled button on the back of the controller. As a last resort (thinking it was a reset button), I pressed and held it for 20 seconds, waited a minute, then briefly pressed it for 1 second while it was out of the unit.
  3. I removed the Compact Flash card from the controller for 1 minute, then re-inserted it. In hoping this would fail the cache copy if it was stuck in the process of writing cache to compact flash.

I then re-inserted the controller, and it booted fine! It was not functioning and working (and came up very fast). Looking at the logs, it has no record of what occurred between the first shutdown, and final boot. I hope this post helps someone else with the same issue, it can save you a support ticket, and time with a controller down.

Disclaimer

PLEASE NOTE: I could not find any information on the unlabeled button on the controller, and it’s hard to know exactly what it does. Perform this at your own risk (make sure you have a backup). Since I have 2 controllers, and my MSA 2040 was running fine on Controller A, I felt comfortable doing this, as if this did reset controller B, the configuration would replicate back from controller A. I would not do this in a single controller environment.

Update – 24 Hours later

After I got everything up and running, I checked the logs of the unit and couldn’t find anything on controller B that looked out of ordinary. However, 24 hours later, I logged back in and noticed some new events showed up from the day before (from the day I had the issues):

MSA 2040 Code 549

MSA 2040 Code 549

You’ll notice the event log with severity error:

Recovery from internal processor fault detected on controller.
Code 549

One thing that’s very odd is that I know for a fact the time is wrong on the error severity log entry, this could be due to the fact we had a daylight savings time change last night at midnight. Either way it appears that it finally did detect that the Storage controller was in an error state and logged it, but it still would have been nice for some more information.

On a final note, the unit has been running perfectly for over 24 hours.

Oct 282018
 

I have noticed an issue when after upgrading Microsoft Exchange 2016 CU10 to Exchange 2016 CU11, services may fail to start. This issue can be intermittent, where some restarts are able to start more services, and others restarts fewer. I have observed this on 2 separate Exchange upgrades, both were CU10 to CU11.

The Problem

Recently, a customer had an issue where a Microsoft Exchange security update bricked their entire Exchange CU10 installation. Files were missing and services would not start (even after manually re-configuring all system services to their prior settings, and force starting). To fix this, we weighed our options and decided the best course of action would be to attempt the latest CU (CU11). This is because each Microsoft Exchange Cumulative update is actually a full installer that completely removes the old version, and installs the new version cleanly.

After installing CU11 we were able to rescue the Exchange installation (services could now start, and functioned), however numerous errors and warnings were now present, and we also noticed that there were some new issues with services.

One service in particular called “Net.Tcp Port Sharing Service”, would occasionally not start in time and cause all the Exchange Services not to start (Exchange is dependent on this services). Other times, this service would start, however random Exchange services would timeout.

Some of the errors and warnings included:

Event ID 7000
Source: Service Control Manager
Description:
The MSComplianceAudit service failed to start due to the following error: 
The service did not respond to the start or control request in a timely fashion.

Event ID 7009
Source: Service Control Manager
Description:
A timeout was reached (30000 milliseconds) while waiting for the MSComplianceAudit service to connect.

Event ID 7000
Source: Service Control Manager
Description:
The MSExchangeRepl service failed to start due to the following error: 
The service did not respond to the start or control request in a timely fashion.

Event ID 7009
Source: Service Control Manager
Description:
A timeout was reached (30000 milliseconds) while waiting for the MSExchangeRepl service to connect.

 

I also observed that on a few restarts, the services that failed would eventually end up restarting 10-15 minutes later (this only occurred 50% of the time).

Originally I was concerned and believed these issues were related to the original issues the customer experienced, however I upgraded my own Exchange 2016 server to CU11 and experienced the same problems (my instance was a clean fully functioning install). I also attempted to upgrade .NET to version 4.7.2 to see if this had any effect, but it did not.

When you go in to services (services.msc) and manually start the services, Exchange functions perfectly and everything works.

The Solution

As of yet, I don’t have a proper solution. I did however notice that with my customer’s environment, after it was left to sit overnight (around 8 hours), that subsequent restarts actually were able to start the majority of the services properly. It almost seemed as if it just needed time to fix itself. I’m not sure if this is because of IO load, or some type of Exchange database maintenance, but I’m waiting to see if it clears up on my instance as well after an amount of time. I’ll be keeping this post updated.

UPDATE – October 29th: I’ve confirmed for the 2nd time that the issue resolves at least 6-8 hours after the upgrade. At the end of the day I restarted my machine and everything was functioning properly.

 

If you are experiencing this issue, or can make a comment on it, please leave a comment on this post!

Oct 212018
 

VMware announced the end of availability for the vDP (vSphere Data Protection) product last year. In their release, it was mentioned that no more updates would be issued for the product.

At that time, I recommended to most customers to start planning to upgrade/migrate to a different product. VMware then released some ESXi and vSphere upgrades, but didn’t immediately update the vDP appliance. It appeared as though no more updates would be issued.

Due to the the previous announcement, and a lack of an updated vDP appliance, this increased the urgency of migrating to a different product.

However, just recently I noticed that VMware is still maintaining the vDP appliance for vSphere 6.5, so you can continue to use it while you plan your migration to a new Backup/DR product, and your upgrade to vSphere version 6.7.

 

Please note: I noticed the updates to the vDP appliance are being released later than the vSphere/ESXi updates. I wouldn’t upgrade your ESXi hosts to 6.5 certain update levels until the applicable vDP updated appliance is available for that version.

Oct 212018
 
People Exercising at Gym

I’m going to stray away from the norm on this blog post. While I usually post about technical content, I’m going to jump in to a bit of health, fitness, exercise, and wellness.

As we all know, being a geek, nerd, and/or IT Professional, we have some crazy hours. Whether it’s pulling all-nighters training/learning, working entire weekend shifts to complete big implementation or migrations, or waking up extremely early to support companies in different geographical areas, it’s all extremely hard on the body and doesn’t give us a lot of time for a personal life or to stay healthy with exercise and fitness. This post is also relevant to the typical business professional, anyone who works or spends large amounts of time on the computer, or anyone who just wants to start working out! Think of this post as a beginners guide!

Disclaimer

I’m not a health professional, nor do I consider myself an expert with any of this. I simply did my own research, came up with my own plan, and found my own resources. I have to recommend to consult with a professional before performing any of the suggested acts in this post. You should only exercise if your doctor clears you and says you are healthy enough to exercise. If you have any pre-existing health concerns, make sure you consult with a doctor before exercising.

The Problem

It’s a challenge to come up with a workout plan due to the hours needed, determination, and having absolutely no idea what to do if you’re just starting out. A lot of people are scared because they don’t want to do it alone. Many also are worried and/or embarrassed about their current physical state, or embarrassed how they’d look at the gym (size or strength levels), and not knowing what they are doing.

One of the biggest problems with living and working on the computer or in the datacenter, is health! Specific problems I’ve felt and experienced over the years, include (but are not limited to):

  • Body Posture (Slouching and Hunching over)
  • Hand, Wrist, and Finger fatigue from typing (and/or carpal tunnel syndrome)
  • Back pain (upper and lower)
  • Sciatic Nerve Pain (Absolutely friggin horrible)
  • Inactivity
  • Difficulty Sleeping (problematic sleeping patterns)
  • Mood
  • Weight Gain

Also, lack of exercise and fitness can have serious impacts on mental health as well. This can seriously interfere with your ability to deal with stress, think clearly, perform well at work, and just feel generally well.

I wanted to share some of the things I do, to keep healthy, stay mentally sharp, deal with stress, and feel great!

Preparation

So you’re ready for the next step, now to prepare. Here’s a few important things you’ll either need to know, or to prepare for mentally:

  • Get some exercise clothes. You don’t need to be a fashionista, and who cares if you look good or bad. You do however need some clothes that will avoid chaffing (mostly for runners), and won’t get caught or grab on equipment at the gym.
    • Running – Compression Boxers/Underwear, shorts, and a compression shirt (helps evaporate sweat and manage body temp), and of course running shoes
    • Strength Training – Shorts and a T-shirt or sweatpants and a shirt, and runners. Most gyms require you to change footwear when entering the gym. Wear your outdoor shoes, but bring a pair of runners to change in to.
  • Whether your running in public, or working out at a gym, you need to make sure you DO NOT CARE what anyone thinks of you (either the way you look, or your current body shape/size). Everyone’s there for the same purpose to look better and get healthy.
  • Stretch!
    • Stretch when you wake up
    • Try to have stretched around 30-60 minutes before a run or gym session (if it’s first thing, your morning stretches can count as these)
    • Stretch after running or gym session
    • Stretch before you go to bed
  • Be prepared to get SORE. You will hurt for the first week, possibly second week, and maybe even the third week. This will get better with time, and you just need to stay with your regiment. If you’re strength training, you can add 15-30 minutes of cardio after your workout session to assist with recovery and reduce pain, however don’t do anymore than 30 minutes as this can affect muscle growth and your gains.
  • Commit to your plan
    • Instead of doing a month-to-month gym membership, try signup on a 1, 2, or 3 year term to make sure you HAVE to go.
    • Get a fitness tracker to track your runs, bike rides, and gym sessions. Visibly seeing what you’re doing allows the competitiveness in you to compete against yourself! And it’s fun!

The Solution

First and foremost, you need to find or make time. If you’re like me and are an early riser, wake up even earlier to go for a run or hit the gym. You can also do so during breaks in the day, lunch for example. If you can’t wake up earlier, then plan for at the end of your work day.

I have a membership at Anytime Fitness, which is a 24/7 gym. Not only are they open 24/7/365 so you can workout whenever you want, but you also get access to their vast network of gym’s worldwide. This is perfect for the corporate traveler, or the individual with crazy hours.

For the two previous years I focused on cardio, however this year I decided to get back in to daily strength training (which I haven’t done for 4 years).

PLEASE NOTE: I’ve been active for the past 4 years, so don’t expect to be able to do the same distances running, distances cycling, or workout plans I do. Please use the resources in this post as a guide to develop your own plan and what you’re comfortable with.

Aerobic (Cardio)

Aerobic exercise is the act in which your muscles use oxygen from the air to sustain a certain activity. This is commonly referred to as cardiovascular exercise, or simply cardio. Below are are activities/exercises I do that involve aerobic cardiovascular exercise.

Biking/Cycling

Cycling for FitnessBike riding is an excellent way to get exercise. You can use any type of bike, get out, see nature, and get a great workout. I have both a mountain bike, and a road bike. Typically I use the mountain bike for outdoor terrain (dirt, paths, etc.), whereas the road bike is used for high-speed, long distance riding and training (pathways, roads, etc).

When I was running daily, I would regularly switch randomly from running to cycling every few days to maintain performance and switch things up. You can burn a crazy amount of calories on extremely long rides, I was recording over 2,000 calories burned on rides over 70km. Occasionally when I wanted to increase distance, I would cut running, and focus primarily on cycling for a few days or a week at a time to crank out the extra distance.

Running

Running has turned in to a personal favorite activity of mine. It helps you get out, unwind, de-stress, think, and just clock-out. There’s not much to running except to throw on some clothes, throw on the running shoes, and your good to go.Running

When I was actively running, I would make sure to run daily. Typically I would shoot for 8-14km per day, and once every week or so I would have a good long 20km+ run. Whenever I noticed I was burning out or performance was plateauing, I would swap some of the daily runs for long bike rides which would then help me realize gains running. The muscle gains from cycling or strength training can help with running further and longer.

Please remember to stretch with running to avoid serious injury.

Hiking

Hiking is another great activity for cardiovascular health, and can also help build leg strength. Luckily for me, I live by the rock mountains so I have Banff, Canmore, and Kananaskis Country right at my doorstep within an hour drive. You can do easy hikes or hard hikes, whatever you’re comfortable with.

Stephen Wagner Hiking

Stephen Wagner Hiking

I like hiking particularly because it’s challenging with the terrain, and you can do it with other people. Once you start to go on more aggressive hikes, you can climb to the top of mountains, take beautiful videos and pictures, and see things most other people will never see in their lifetimes.

Anaerobic (Strength Training)

Anaerobic exercise is the act in which your body uses glycogen and fat as fuel to sustain an activity. This is commonly referred to as strength training exercises, or simply lifting weights. Below is how I perform Anaerobic strength training exercises.

Strength Training
Strength Training Using Weight Machine

Strength Training Using Weight Machines

As mentioned above, I’m a member at Anytime Fitness. I usually wake up at around 3:00AM to 4:30AM everyday, work on e-mails and overnight issues, and then visit the gym at around 5:00AM to 6:00AM. It’s the perfect routine for me as it allows me to deal with emergencies and kick off the day with a great workout (which gives me crazy amounts of energy). You can do a similar schedule, or find one that works best for you!

For work out plans, you can find many on the website www.bodybuilding.com. They have a wide variety of content for the different type of workouts you may want to do, you can also click/tap on the specific exercises in the workout plans, view videos of the exercises and learn how to do them, as well as read on proper form an technique to make sure you’re doing it right. Having your phone with you while you’re at the gym allows you to follow this exactly and make sure you are actually doing it right to get the most out of the exercise as well as avoid injury.

The current 5-day workout plan I’m on is “J-DAWG”‘s at: https://www.bodybuilding.com/content/what-is-the-best-5-day-workout-split.html

If you’re just starting out, I might recommend a plan similar to this one: https://www.bodybuilding.com/content/an-amazing-4-day-workout-for-lean-mass.html.

If you’re on an aggressive plan, make sure to have 1-2 rest days every 7 days, even more if you’re just starting out. You should never work the same muscle group more than once in 24 hours.

If you’re just starting out, maybe try using the weight machines more, then as you’re comfortable move to the free weights. Free weights are ultimately what you want to be using in the majority of your exercises as they not only work on your main muscle groups, but also develop stabilizing muscles.

Strength Training using Free Weights

Strength Training using Free Weights

Make it fun/Extra Tips/Supplements

There’s a few extra tips I want to provide to help out as well as make it more fun!

Energy Boost

On those crazy days where you didn’t get enough sleep, or you’re feeling drained, hitting the gym first thing can give you a massive burst of energy… And if you start dying out in the afternoon, a quick 30-45 minute run can re-inject you with another 4-6 hours of energy to help you finish your day. After a night of no sleep, or a night out partying, I commonly use this method to help me get through the next day!

Fitness Trackers

One of the great ways to stay motivated is to buy a fitness tracker (smartwatch) that can track your progress. Not only can you prove you worked out that day to self-motivate, but you can also track your progress, vitals, and how your body is performing. I’ve used a few including the Microsoft Band, Microsoft Band 2, and now I’m using the Samsung Gear S3. The Samsung Gear S3 is my favorite.

Samsung Gear S3

Samsung Gear S3

With my Samsung Gear S3, when I’m running or cycling I can track the route with GPS, minutes per KM/Mile, heart rate, and other metrics. When my Gear S3 syncs with Samsung Health, it can also sync with Strava. Strava is an online platform that tracks Running and Cycling over a social network at https://www.strava.com.

The Strava App

The Strava App

Strava is an awesome way to track your progress and connect with your friends who are also on the platform.

When it comes to strength training, my gym Anytime Fitness, has an app for my phone. The Anytime Fitness app tracks you gym visits, connects with the Under Armor “Map My Run” app (despite run in the name, it can also track strength training), and allows you to track your burn, heart rate, and other information. For strength training I typically use the Under Armor app, as it connects to the Anytime Fitness app. However, you can use the standard Samsung Health app on our Samsung Gear S3 to track strength training as well.

Supplements

It’s all about the protein! Whether I’m doing cardio or strength training, I consume as much as I can of this stuff as I can. I find when taking 1.5 hours before a run, I can run longer with more energy (always take some after as well). When it comes to strength training, protein is extremely important for muscle growth, and to stop your body from eating away at them (which happens if you don’t get enough protein). Once you start advanced training, you can add more supplements as needed.

I purchase most of my supplements from www.bodybuilding.com, or the local suppliment store. The website bodybuilding.com also has a great protein dosage calculator at: https://www.bodybuilding.com/fun/calpro.htm

 

I hope the information in this post either helps your existing workouts, or motivates you to get on the fitness bandwagon. Please feel free to leave a comment and correct me on anything, or add your own advice. I will occasionally update this post, so feel free to bookmark and check from time to time in case I post anything new!

Oct 202018
 

On an ESXi host when performing a manual unmap on your storage datastore, you may notice a very large (hidden) file on the datastore root called “.asyncUnmapFile”. This file could be taking up terabytes of space, and you aren’t able to delete this file.

asyncUnmapFile

Typically the asyncUnmapFile is used by the UNMAP feature on ESXi hosts to deal with unmapping and unallocating storage blocks on the SAN. When you run a manual UNMAP, this file should be created and should appear to using “0” (no) space (even if it is). When an UNMAP completes, this file should disappear and be automatically removed by the function. If an UNMAP is interupted, this file will not be deleted, allowing you to restart the process and upon a full successful completion, it should then be deleted.

The Problem

Some time ago, I had an issue when performing a manual UNMAP, where the ESXi host became unresponsive (due to memory issues). The command appeared to be completed, however I believe it caused potential issues or corruption on the iSCSI datastore. In subsequent runs, the UNMAP appeared to be functioning and working, however I didn’t realize that the asyncUnmapFile had grown to around 1.5TB.

This was noticed during a SAN storage audit, where we saw that the virtual pool on the SAN was using up way more storage than it should be on the datastore.

When we identified the file was this large and causing issues, we attempted to perform 2 UNMAPs (different reclaim sizes) to see if it would be automatically cleared afterwards. It had no effect and the file was unchanged.

We also tried to modify the permissions on the file, however when trying to delete it, it would report that the file or folder was not found, or that it does not exist. This was concerning as we were worried about potential datastore corruption.

It was also noticed that in the hostd.log and vmkernel.log we saw some errors where the host believed that the blocks on the datastore had already been freed: “on volume labeled ‘iSCSIDatastore01’ already freed by another host: This may be a non-issue”

The Solution

Unfortunately with all the research we did, we couldn’t find a clear-cut solution. With worries that the datastore may be corrupted, we needed to do something.

A decision was ultimately made to Storage vMotion all the VMs (Virtual Machines) to another datastore on a separate storage pool, delete the now empty LUN, and recreate it from scratch. After this, we used Storage vMotion again to move the VMs back.

Instantly I noticed that the VMs on that datastore were running faster (it’s only been 12 hours, so I’ll be adding an update in a few days to confirm). We no longer have the file on any of our datastores.

If anyone has further insight in to this issue, please leave a comment!

Oct 162018
 

In this post, I’ll be going over how to add additional and/or alternative UPN suffixes to your Active Directory. I’ll also be going over why you may require this inside of your environment.

This is also a follow up post to the article here: https://www.stephenwagner.com/2016/09/23/outlook-2016-exchange-2013-password-prompts-upn-and-samaccountname-troubles/ as Microsoft has deleted the KB 243629 article which contained the original instructions.

Why

There is a number of reasons why you may want to do this:

  1. You’re migrating to a newer version of Microsoft Outlook 2016, and require the users UPN to match the users e-mail address for auto-configure to function.
  2. Your internal domain is is a “domain.local” domain, however you want users to log in with a “domain.com” domain.
  3. You are implementing a line of business application or other piece of software that requires user’s UPNs to match their e-mail addresses.
  4. You’re performing a migration.

How

Let’s get to it! Here’s how to add an alternative UPN suffix to an Active Directory domain:

  1. Log on to your domain controller.
  2. Open “Active Directory Domains and Trusts”
  3. On the left hand side of the new window, right click on “Active Directory Domains and Trusts”, and select “Properties” (as shown below).
    Active Directory Domains and Trusts Window

    Active Directory Domains and Trusts Window

     

  4. Type in your new domain suffix in to the “Alternative UPN suffixes” box, and then click “Add”. As shown below.
    Add Alternative UPN suffix

    Add Alternative UPN suffix

     

  5. Click “Apply” and then close out of the windows.

The new UPN suffix should be available via “Active Directory Users and Computers” and you should be able to set it to users.

You can also configure the user accounts via the Exchange Administration Center (EAC). See below for an example:

Exchange Administration Center UPN Suffix

Exchange Administration Center UPN Suffix

 

Oct 122018
 
DNS

In the perfect and properly configured world, every internet user has a reverse DNS entry. This is is the DNS entry which tells people, servers, and services, what any given IP’s hostname is. Also, again in the perfect world, web servers shouldn’t check these, as the DNS query itself usually has to complete before it starts serving website data.

One of the key way’s webmasters and web server administrators increase their web server response times, is to make sure that their server is NOT performing reverse DNS queries when serving the site. However, we aren’t in a perfect world, and many web servers and web sites still perform these queries.

Many web servers do these queries because they are using mis-configured statistic generation software (website stats), default web server configuration files, or other reasons.

The problem

I recently had a discussion with a fellow IT professional where they were having issues with load times when opening websites. They were on a high speed business internet connection, so they figured it had to do with something else. They said they checked absolutely everything, so I decided to see what I could do to help out!

In my own research I noticed that on my own web server (which doesn’t perform reverse DNS queries on users), that numerous visitors both local to North America and abroad, did not actually have properly configured reverse DNS entries. One can deduce that when one of these users visits a website that actually performs an RDNS check during initial connection, it could cause a delay while the server itself waits for the DNS query to be performed (or even worse, timeout).

When further investigating, I also noticed a trend that the larger the company and the more expensive the internet connection, the more IPs that did not have reverse DNS records. I also noticed the IP addresses provided to my colleague did not have RDNS records.

I relayed this information back to my colleague and after they created the proper reverse DNS records, it seemed to help the issue!

Final Note

Since I don’t have direct access to their network, I couldn’t confirm this was the actual issue, or the only issue, but this just goes to show that you should always have your networks (both internal and external) properly configured using leading practices. In the long run, it saves time and avoids issues.

Oct 082018
 
Microsoft Windows Logo

If you are running Microsoft Windows in a domain environment with WSUS configured, you may notice that you’re not able to install some FODs (Features on Demand), or use the “Turn Windows features on or off”. This will stop you from installing things like the RSAT tools, .NET Framework, Language Speech packs, etc…

You may see “failure to download files”, “cannot download”, or errors like “0x800F0954” when running DISM to install packages.

To resolve this, you need to modify your domain’s group policy settings to allow your workstations to query Windows Update servers for additional content. The workstations will still use your WSUS server for approvals, downloads, and updates, however in the event content is not found, it will query Windows Update.

Enable download of “Optional features” directly from Windows Update

  1. Open the group policy editor on your domain
  2. Create a new GPO, or modify an existing one. Make sure it applies to the computers you’d like
  3. Navigate to “Computer Configuration”, “Policies”, “Administrative Templates”, and then “System”.
  4. Double click or open “Specify settings for optional component installation and component repair”
  5. Make sure “Never attempt to download payload from Windows Update” is NOT checked
  6. Make sure “Download repair content and optional features directly from Windows Update instead of Windows Server Update Services (WSUS)” IS checked.
  7. Wait for your GPO to update, or run “gpupdate /force” on the workstations.

Please see an example of the configuration below:

Download repair content and optional features directly from Windows Update instead of Windows Server Update Services (WSUS)

You should now be able to download/install RSAT, .NET, Speech language packs, and more!

Oct 072018
 
Microsoft Windows Logo

Just a few words of warning when upgrading your VMware vSphere Windows 10 virtual machines to Windows 10 Version 1809 (October Update). When upgrading, after the first restart, you may notice multiple BSOD (Blue Screen of Death) with the error “Driver PNP Watchdog”. This will fail the upgrade.

Update – November 14 2018: This issue is still occurring on upgrades using the re-released November version of the October update.

When the upgrade fails, the system will re-attempt until utlimately failing and reverting to the previous version of Windows 10.

In my case, I had a successful upgrade on numerous physical workstations, and a snapshot, so I decided to uninstall both the VMware tools agent, and VMware Horizon View agent. This had no affect and the VM still wouldn’t perform an upgrade.

I’m not sure if it’s the fact that it’s a VM, the VMware tools install, or the VMware Horizon View agent install, however I highly recommend waiting to upgrade until all the bugs get sorted out.

Leave a comment if you have anything to add! 🙂