Oct 282018
 

I have noticed an issue when after upgrading Microsoft Exchange 2016 CU10 to Exchange 2016 CU11, services may fail to start. This issue can be intermittent, where some restarts are able to start more services, and others restarts fewer. I have observed this on 2 separate Exchange upgrades, both were CU10 to CU11.

The Problem

Recently, a customer had an issue where a Microsoft Exchange security update bricked their entire Exchange CU10 installation. Files were missing and services would not start (even after manually re-configuring all system services to their prior settings, and force starting). To fix this, we weighed our options and decided the best course of action would be to attempt the latest CU (CU11). This is because each Microsoft Exchange Cumulative update is actually a full installer that completely removes the old version, and installs the new version cleanly.

After installing CU11 we were able to rescue the Exchange installation (services could now start, and functioned), however numerous errors and warnings were now present, and we also noticed that there were some new issues with services.

One service in particular called “Net.Tcp Port Sharing Service”, would occasionally not start in time and cause all the Exchange Services not to start (Exchange is dependent on this services). Other times, this service would start, however random Exchange services would timeout.

Some of the errors and warnings included:

Event ID 7000
Source: Service Control Manager
Description:
The MSComplianceAudit service failed to start due to the following error: 
The service did not respond to the start or control request in a timely fashion.

Event ID 7009
Source: Service Control Manager
Description:
A timeout was reached (30000 milliseconds) while waiting for the MSComplianceAudit service to connect.

Event ID 7000
Source: Service Control Manager
Description:
The MSExchangeRepl service failed to start due to the following error: 
The service did not respond to the start or control request in a timely fashion.

Event ID 7009
Source: Service Control Manager
Description:
A timeout was reached (30000 milliseconds) while waiting for the MSExchangeRepl service to connect.

I also observed that on a few restarts, the services that failed would eventually end up restarting 10-15 minutes later (this only occurred 50% of the time).

Originally I was concerned and believed these issues were related to the original issues the customer experienced, however I upgraded my own Exchange 2016 server to CU11 and experienced the same problems (my instance was a clean fully functioning install). I also attempted to upgrade .NET to version 4.7.2 to see if this had any effect, but it did not.

When you go in to services (services.msc) and manually start the services, Exchange functions perfectly and everything works.

The Solution

As of yet, I don’t have a proper solution. I did however notice that with my customer’s environment, after it was left to sit overnight (around 8 hours), that subsequent restarts actually were able to start the majority of the services properly. It almost seemed as if it just needed time to fix itself. I’m not sure if this is because of IO load, or some type of Exchange database maintenance, but I’m waiting to see if it clears up on my instance as well after an amount of time. I’ll be keeping this post updated.

UPDATE – October 29th: I’ve confirmed for the 2nd time that the issue resolves at least 6-8 hours after the upgrade. At the end of the day I restarted my machine and everything was functioning properly.

If you are experiencing this issue, or can make a comment on it, please leave a comment on this post!

Additional Resources

Oct 212018
 

VMware announced the end of availability for the vDP (vSphere Data Protection) product last year. In their release, it was mentioned that no more updates would be issued for the product.

At that time, I recommended to most customers to start planning to upgrade/migrate to a different product. VMware then released some ESXi and vSphere upgrades, but didn’t immediately update the vDP appliance. It appeared as though no more updates would be issued.

Due to the the previous announcement, and a lack of an updated vDP appliance, this increased the urgency of migrating to a different product.

However, just recently I noticed that VMware is still maintaining the vDP appliance for vSphere 6.5, so you can continue to use it while you plan your migration to a new Backup/DR product, and your upgrade to vSphere version 6.7.

 

Please note: I noticed the updates to the vDP appliance are being released later than the vSphere/ESXi updates. I wouldn’t upgrade your ESXi hosts to 6.5 certain update levels until the applicable vDP updated appliance is available for that version.

Oct 212018
 
People Exercising at Gym

I’m going to stray away from the norm on this blog post. While I usually post about technical content, I’m going to jump in to a bit of health, fitness, exercise, and wellness.

As we all know, being a geek, nerd, and/or IT Professional, we have some crazy hours. Whether it’s pulling all-nighters training/learning, working entire weekend shifts to complete big implementation or migrations, or waking up extremely early to support companies in different geographical areas, it’s all extremely hard on the body and doesn’t give us a lot of time for a personal life or to stay healthy with exercise and fitness. This post is also relevant to the typical business professional, anyone who works or spends large amounts of time on the computer, or anyone who just wants to start working out! Think of this post as a beginners guide!

Disclaimer

I’m not a health professional, nor do I consider myself an expert with any of this. I simply did my own research, came up with my own plan, and found my own resources. I have to recommend to consult with a professional before performing any of the suggested acts in this post. You should only exercise if your doctor clears you and says you are healthy enough to exercise. If you have any pre-existing health concerns, make sure you consult with a doctor before exercising.

The Problem

It’s a challenge to come up with a workout plan due to the hours needed, determination, and having absolutely no idea what to do if you’re just starting out. A lot of people are scared because they don’t want to do it alone. Many also are worried and/or embarrassed about their current physical state, or embarrassed how they’d look at the gym (size or strength levels), and not knowing what they are doing.

One of the biggest problems with living and working on the computer or in the datacenter, is health! Specific problems I’ve felt and experienced over the years, include (but are not limited to):

  • Body Posture (Slouching and Hunching over)
  • Hand, Wrist, and Finger fatigue from typing (and/or carpal tunnel syndrome)
  • Back pain (upper and lower)
  • Sciatic Nerve Pain (Absolutely friggin horrible)
  • Inactivity
  • Difficulty Sleeping (problematic sleeping patterns)
  • Mood
  • Weight Gain

Also, lack of exercise and fitness can have serious impacts on mental health as well. This can seriously interfere with your ability to deal with stress, think clearly, perform well at work, and just feel generally well.

I wanted to share some of the things I do, to keep healthy, stay mentally sharp, deal with stress, and feel great!

Preparation

So you’re ready for the next step, now to prepare. Here’s a few important things you’ll either need to know, or to prepare for mentally:

  • Get some exercise clothes. You don’t need to be a fashionista, and who cares if you look good or bad. You do however need some clothes that will avoid chaffing (mostly for runners), and won’t get caught or grab on equipment at the gym.
    • Running – Compression Boxers/Underwear, shorts, and a compression shirt (helps evaporate sweat and manage body temp), and of course running shoes
    • Strength Training – Shorts and a T-shirt or sweatpants and a shirt, and runners. Most gyms require you to change footwear when entering the gym. Wear your outdoor shoes, but bring a pair of runners to change in to.
  • Whether your running in public, or working out at a gym, you need to make sure you DO NOT CARE what anyone thinks of you (either the way you look, or your current body shape/size). Everyone’s there for the same purpose to look better and get healthy.
  • Stretch!
    • Stretch when you wake up
    • Try to have stretched around 30-60 minutes before a run or gym session (if it’s first thing, your morning stretches can count as these)
    • Stretch after running or gym session
    • Stretch before you go to bed
  • Be prepared to get SORE. You will hurt for the first week, possibly second week, and maybe even the third week. This will get better with time, and you just need to stay with your regiment. If you’re strength training, you can add 15-30 minutes of cardio after your workout session to assist with recovery and reduce pain, however don’t do anymore than 30 minutes as this can affect muscle growth and your gains.
  • Commit to your plan
    • Instead of doing a month-to-month gym membership, try signup on a 1, 2, or 3 year term to make sure you HAVE to go.
    • Get a fitness tracker to track your runs, bike rides, and gym sessions. Visibly seeing what you’re doing allows the competitiveness in you to compete against yourself! And it’s fun!

The Solution

First and foremost, you need to find or make time. If you’re like me and are an early riser, wake up even earlier to go for a run or hit the gym. You can also do so during breaks in the day, lunch for example. If you can’t wake up earlier, then plan for at the end of your work day.

I have a membership at Anytime Fitness, which is a 24/7 gym. Not only are they open 24/7/365 so you can workout whenever you want, but you also get access to their vast network of gym’s worldwide. This is perfect for the corporate traveler, or the individual with crazy hours.

For the two previous years I focused on cardio, however this year I decided to get back in to daily strength training (which I haven’t done for 4 years).

PLEASE NOTE: I’ve been active for the past 4 years, so don’t expect to be able to do the same distances running, distances cycling, or workout plans I do. Please use the resources in this post as a guide to develop your own plan and what you’re comfortable with.

Aerobic (Cardio)

Aerobic exercise is the act in which your muscles use oxygen from the air to sustain a certain activity. This is commonly referred to as cardiovascular exercise, or simply cardio. Below are are activities/exercises I do that involve aerobic cardiovascular exercise.

Biking/Cycling

Cycling for FitnessBike riding is an excellent way to get exercise. You can use any type of bike, get out, see nature, and get a great workout. I have both a mountain bike, and a road bike. Typically I use the mountain bike for outdoor terrain (dirt, paths, etc.), whereas the road bike is used for high-speed, long distance riding and training (pathways, roads, etc).

When I was running daily, I would regularly switch randomly from running to cycling every few days to maintain performance and switch things up. You can burn a crazy amount of calories on extremely long rides, I was recording over 2,000 calories burned on rides over 70km. Occasionally when I wanted to increase distance, I would cut running, and focus primarily on cycling for a few days or a week at a time to crank out the extra distance.

Running

Running has turned in to a personal favorite activity of mine. It helps you get out, unwind, de-stress, think, and just clock-out. There’s not much to running except to throw on some clothes, throw on the running shoes, and your good to go.Running

When I was actively running, I would make sure to run daily. Typically I would shoot for 8-14km per day, and once every week or so I would have a good long 20km+ run. Whenever I noticed I was burning out or performance was plateauing, I would swap some of the daily runs for long bike rides which would then help me realize gains running. The muscle gains from cycling or strength training can help with running further and longer.

Please remember to stretch with running to avoid serious injury.

Hiking

Hiking is another great activity for cardiovascular health, and can also help build leg strength. Luckily for me, I live by the rock mountains so I have Banff, Canmore, and Kananaskis Country right at my doorstep within an hour drive. You can do easy hikes or hard hikes, whatever you’re comfortable with.

Stephen Wagner Hiking

Stephen Wagner Hiking

I like hiking particularly because it’s challenging with the terrain, and you can do it with other people. Once you start to go on more aggressive hikes, you can climb to the top of mountains, take beautiful videos and pictures, and see things most other people will never see in their lifetimes.

Anaerobic (Strength Training)

Anaerobic exercise is the act in which your body uses glycogen and fat as fuel to sustain an activity. This is commonly referred to as strength training exercises, or simply lifting weights. Below is how I perform Anaerobic strength training exercises.

Strength Training
Strength Training Using Weight Machine

Strength Training Using Weight Machines

As mentioned above, I’m a member at Anytime Fitness. I usually wake up at around 3:00AM to 4:30AM everyday, work on e-mails and overnight issues, and then visit the gym at around 5:00AM to 6:00AM. It’s the perfect routine for me as it allows me to deal with emergencies and kick off the day with a great workout (which gives me crazy amounts of energy). You can do a similar schedule, or find one that works best for you!

For work out plans, you can find many on the website www.bodybuilding.com. They have a wide variety of content for the different type of workouts you may want to do, you can also click/tap on the specific exercises in the workout plans, view videos of the exercises and learn how to do them, as well as read on proper form an technique to make sure you’re doing it right. Having your phone with you while you’re at the gym allows you to follow this exactly and make sure you are actually doing it right to get the most out of the exercise as well as avoid injury.

The current 5-day workout plan I’m on is “J-DAWG”‘s at: https://www.bodybuilding.com/content/what-is-the-best-5-day-workout-split.html

If you’re just starting out, I might recommend a plan similar to this one: https://www.bodybuilding.com/content/an-amazing-4-day-workout-for-lean-mass.html.

If you’re on an aggressive plan, make sure to have 1-2 rest days every 7 days, even more if you’re just starting out. You should never work the same muscle group more than once in 24 hours.

If you’re just starting out, maybe try using the weight machines more, then as you’re comfortable move to the free weights. Free weights are ultimately what you want to be using in the majority of your exercises as they not only work on your main muscle groups, but also develop stabilizing muscles.

Strength Training using Free Weights

Strength Training using Free Weights

Make it fun/Extra Tips/Supplements

There’s a few extra tips I want to provide to help out as well as make it more fun!

Energy Boost

On those crazy days where you didn’t get enough sleep, or you’re feeling drained, hitting the gym first thing can give you a massive burst of energy… And if you start dying out in the afternoon, a quick 30-45 minute run can re-inject you with another 4-6 hours of energy to help you finish your day. After a night of no sleep, or a night out partying, I commonly use this method to help me get through the next day!

Fitness Trackers

One of the great ways to stay motivated is to buy a fitness tracker (smartwatch) that can track your progress. Not only can you prove you worked out that day to self-motivate, but you can also track your progress, vitals, and how your body is performing. I’ve used a few including the Microsoft Band, Microsoft Band 2, and now I’m using the Samsung Gear S3. The Samsung Gear S3 is my favorite.

Samsung Gear S3

Samsung Gear S3

With my Samsung Gear S3, when I’m running or cycling I can track the route with GPS, minutes per KM/Mile, heart rate, and other metrics. When my Gear S3 syncs with Samsung Health, it can also sync with Strava. Strava is an online platform that tracks Running and Cycling over a social network at https://www.strava.com.

The Strava App

The Strava App

Strava is an awesome way to track your progress and connect with your friends who are also on the platform.

When it comes to strength training, my gym Anytime Fitness, has an app for my phone. The Anytime Fitness app tracks you gym visits, connects with the Under Armor “Map My Run” app (despite run in the name, it can also track strength training), and allows you to track your burn, heart rate, and other information. For strength training I typically use the Under Armor app, as it connects to the Anytime Fitness app. However, you can use the standard Samsung Health app on our Samsung Gear S3 to track strength training as well.

Supplements

It’s all about the protein! Whether I’m doing cardio or strength training, I consume as much as I can of this stuff as I can. I find when taking 1.5 hours before a run, I can run longer with more energy (always take some after as well). When it comes to strength training, protein is extremely important for muscle growth, and to stop your body from eating away at them (which happens if you don’t get enough protein). Once you start advanced training, you can add more supplements as needed.

I purchase most of my supplements from www.bodybuilding.com, or the local suppliment store. The website bodybuilding.com also has a great protein dosage calculator at: https://www.bodybuilding.com/fun/calpro.htm

 

I hope the information in this post either helps your existing workouts, or motivates you to get on the fitness bandwagon. Please feel free to leave a comment and correct me on anything, or add your own advice. I will occasionally update this post, so feel free to bookmark and check from time to time in case I post anything new!

Oct 202018
 

On an ESXi host when performing a manual unmap on your storage datastore, you may notice a very large (hidden) file on the datastore root called “.asyncUnmapFile”. This file could be taking up terabytes of space, and you aren’t able to delete this file.

asyncUnmapFile

Typically the asyncUnmapFile is used by the UNMAP feature on ESXi hosts to deal with unmapping and unallocating storage blocks on the SAN. When you run a manual UNMAP, this file should be created and should appear to using “0” (no) space (even if it is). When an UNMAP completes, this file should disappear and be automatically removed by the function. If an UNMAP is interupted, this file will not be deleted, allowing you to restart the process and upon a full successful completion, it should then be deleted.

The Problem

Some time ago, I had an issue when performing a manual UNMAP, where the ESXi host became unresponsive (due to memory issues). The command appeared to be completed, however I believe it caused potential issues or corruption on the iSCSI datastore. In subsequent runs, the UNMAP appeared to be functioning and working, however I didn’t realize that the asyncUnmapFile had grown to around 1.5TB.

This was noticed during a SAN storage audit, where we saw that the virtual pool on the SAN was using up way more storage than it should be on the datastore.

When we identified the file was this large and causing issues, we attempted to perform 2 UNMAPs (different reclaim sizes) to see if it would be automatically cleared afterwards. It had no effect and the file was unchanged.

We also tried to modify the permissions on the file, however when trying to delete it, it would report that the file or folder was not found, or that it does not exist. This was concerning as we were worried about potential datastore corruption.

It was also noticed that in the hostd.log and vmkernel.log we saw some errors where the host believed that the blocks on the datastore had already been freed: “on volume labeled ‘iSCSIDatastore01’ already freed by another host: This may be a non-issue”

The Solution

Unfortunately with all the research we did, we couldn’t find a clear-cut solution. With worries that the datastore may be corrupted, we needed to do something.

A decision was ultimately made to Storage vMotion all the VMs (Virtual Machines) to another datastore on a separate storage pool, delete the now empty LUN, and recreate it from scratch. After this, we used Storage vMotion again to move the VMs back.

Instantly I noticed that the VMs on that datastore were running faster (it’s only been 12 hours, so I’ll be adding an update in a few days to confirm). We no longer have the file on any of our datastores.

If anyone has further insight in to this issue, please leave a comment!

Oct 162018
 

In this post, I’ll be going over how to add additional and/or alternative UPN suffixes to your Active Directory. I’ll also be going over why you may require this inside of your environment.

This is also a follow up post to the article here: https://www.stephenwagner.com/2016/09/23/outlook-2016-exchange-2013-password-prompts-upn-and-samaccountname-troubles/ as Microsoft has deleted the KB 243629 article which contained the original instructions.

Why

There is a number of reasons why you may want to do this:

  1. You’re migrating to a newer version of Microsoft Outlook 2016, and require the users UPN to match the users e-mail address for auto-configure to function.
  2. Your internal domain is is a “domain.local” domain, however you want users to log in with a “domain.com” domain.
  3. You are implementing a line of business application or other piece of software that requires user’s UPNs to match their e-mail addresses.
  4. You’re performing a migration.

How

Let’s get to it! Here’s how to add an alternative UPN suffix to an Active Directory domain:

  1. Log on to your domain controller.
  2. Open “Active Directory Domains and Trusts”
  3. On the left hand side of the new window, right click on “Active Directory Domains and Trusts”, and select “Properties” (as shown below).
    Active Directory Domains and Trusts Window

    Active Directory Domains and Trusts Window

     

  4. Type in your new domain suffix in to the “Alternative UPN suffixes” box, and then click “Add”. As shown below.
    Add Alternative UPN suffix

    Add Alternative UPN suffix

     

  5. Click “Apply” and then close out of the windows.

The new UPN suffix should be available via “Active Directory Users and Computers” and you should be able to set it to users.

You can also configure the user accounts via the Exchange Administration Center (EAC). See below for an example:

Exchange Administration Center UPN Suffix

Exchange Administration Center UPN Suffix