Dec 072016

During my first migration from VMware vCenter 6.0 to VMware vCenter 6.5 Virtual appliance, the migration failed. The migration installation UI would shutdown the source VM, and numerous errors would occur afterwards when the destination vCenter appliance would try finishing configuration.

If you were monitoring the source vCenter server, during the export process, one would notice that an error pops up while compressing the source data. The error presented is generated from Windows creating an archive (zip file), the error reads: “The compressed (zipped) folder is invalid or corrupted.”. The entire migration process halts until you dismiss this message, with the entire migration ultimately failing (at first it appears to continue, but ultimately fails).

If you continued, and had the migration fail. You’ll need to power off the failed (new) vCenter appliance (it’s garbage now), and you’ll need to power on the source (original) vCenter server. The active directory trust will no longer exist at this point, so you’ll need to log on with a local (non-domain) account (on the source server), and re-create the computer trust on the domain using the netdom command:


After re-creating the trust, restart the original vCenter server. You have now reverted to your original vCenter instance and can retry the migration.

Now back to the main issue. I tried a bunch of different things and wasted an entire evening (checking character lengths on paths/filenames, trying different settings, pausing processes in case timeouts were being hit, etc…) however finally I noticed that the compression archive would crash/fail on a file called “vum_registry”.

VUM brings VMware Update Manager to mind, which I do have installed, configured, and running.

I went ahead and uninstalled VMware Update Manager off my source server (as it’s easy enough to re-configure from scratch after the migration). I then proceeded to initiate a migration. To my surprise, the “data to migrate” went from 7.9GB to 2.4GB. This is a huge sign that something was messed up with my VMware update manager deployment (even though it was working fine). I’m assuming there were either filenames that were too long (exceeded the 260 character limit on paths and filenames), special characters were being used where they shouldn’t, or something else was messed up.

After the uninstall of Update Manager, the migration completed successfully. Leave a comment!

Dec 052016

In the process of prepping my test environment so I can upgrade from vSphere 6.1 to 6.5, one of the prerequisites is to first upgrade your VDP appliances to version 6.1.3 (6.1.3 is the only version of VDP that supports vSphere 6.5). In my environment I’ll be upgrading VDP from 6.1.2 to 6.1.3.

After downloading the ISO, changing my disks to dependant, creating a snapshot, and attaching the ISO to the VM. My VDP appliances would not recognize the ISO image, showing the dreaded: “To upgrade your VDP appliance, place connect a valid upgrade ISO image to the appliance.”.


I tried a few things, including trying the old “patch” that was issues for 6.1 when it couldn’t detect, unfortunately it didn’t help. I also tried to manually mount the virtual CD-Rom to the mountpoint but had no luck. The mountpoint /mnt/auto/cdrom is locked by the autofs service. If you try to modify these files (such as delete, create, etc…), you’ll encounter a bunch of errors and have no luck (permission denied, file and/or directory doesn’t exist, etc…).

Essentially the autofs service was not auto-mounting the virtual CD drive to the mount point.

To fix this:

  1. SSH in to the VDP appliance
  2. Run command “sudo su” to run commands as root
  3. Use vi to edit the auto.mnt file using command: “vi /etc/auto.mnt”
  4. At the end of the first line in the file, you will see “/dev/cdrom” (without quotation), change this to “/dev/sr0” (again, without quotation)
  5. Save the file (after editing the text, Ctrl+c, then type “:w” and enter which writes the file, then type “:q” then enter to quit vi.
  6. Reload the autofs config using command: “/etc/init.d/autofs reload”
  7. At the shell, run “mount” to show the active mountpoints, you’ll notice the ISO is now mounted after a few seconds.
  8. You can now initiate the upgrade. Start it.
  9. At 71%, autofs updates via a RPM, and the changes you made to the config are cleared. IMMEDIATELY edit the /etc/auto.mnt file again, change “/dev/cdrom” to “/dev/sr0” and save the file, and issue the command “/etc/init.d/autofs reload”. Do this as fast as possible.
  10. You’re good to go, the install will continue and take some time. The web interface will fail, and become unresponsive. Simply wait, and the vDP appliance will eventually shut down (in my case it took over 30 minutes after the web interface failed to reconnect, in a high performance environment for the vDP VM to shut down).

And done! Leave a comment!


Nov 102016

I recently got my hands on a D-Link DCS-5222L IP Camera and thought it would be pretty cool to get it setup to do YouTube Live streaming. To configure the stream I would have to find a way to convert and encode the RTSP stream from the camera to a YouTube Live compatible RTMP stream.

I figured I would see if I could use VLC or FFMPEG to convert the stream. I had no luck with VLC however I found out ffmpeg should do what I needed to do. I couldn’t easily find the proper usage command and flags, however eventually after viewing multiple sites, mixing, matching, and testing, I found this command worked:

ffmpeg -rtsp_transport tcp -i rtsp://IP.CAM.ADD.RESS/live1.sdp -framerate 30 -video_size 1280×720 -vcodec libx264 -preset veryfast -maxrate 1984k -bufsize 3968k -vf “format=yuv420p” -g 60 -c:a aac -b:a 128k -ar 44100 -f flv rtmp://YOUTUBELIVE/DIR/StreamKey

Happy streaming! And if anyone can refine, or recommend better command usage, please post in the comments!

Nov 052016

Yesterday, I had a reader (Nicolas) leave a comment on one of my previous blog posts bringing my attention to the MTU for Jumbo Frames on the HPe MSA 2040 SAN.

MSA 2040 MTU Comment









Since I first started working with the MSA 2040. Looking at numerous HPe documents outlining configuration and best practices, the documents did confirm that the unit supported Jumbo Frames. However, the documentation on the MTU was never clearly stated and can be confusing. I was under the assumption that the unit supported 9000 MTU, while reserving 100 bytes for overhead. This is not necessarily the case.

Nicolas chimed in and provided details on his tests which confirmed the HPe MSA 2040 does actually have a working MTU of 8900. In my configuration I did the tests (that Nicolas outlined), and confirmed that the MTU would cause packet fragmentation if the MTU was greater than 8900.

ESXi vmkping usage:

This is a big discovery because packet fragmentation will not only degrade performance, but flood the links with lots of packet fragmentation.

I went ahead and re-configured my ESXi hosts to use an MTU of 8900 on the network used with my SAN. This immediately created a MASSIVE performance increase (both speed, and IOPS). I highly recommend that users of the MSA 2040 SAN confirm this on their own, and update the MTUs as they see fit.

Also, this brings up another consideration. Ideally, on a single network, you want all devices to be running the same MTU. If your MSA 2040 SAN is on a storage network with other SAN devices (or any other device), you may want to configure all of them to use the MTU of 8900 if possible (and of course, don’t forget your servers).

A big thank you to Nicolas for pointing this out!

Sep 232016

There’s quite a few of us that started off deploying Small Business Server (SBS2008, SBS2011) environments back in the day, loving the handy all-in-one package taking care of everything from Active Directory and Exchange, to disaster recovery and business continuity. However, some of these old environments are starting to catch up with us. I wanted to open a discussion on a big issue I had a couple years ago in one of my first migrations from SBS 2008, to Windows Server 2012 R2 with the Essentials Experience role installed, with Exchange Server 2013.

As most of you know, SBS comes packaged to push “.local” domains on initial domain configuration. This used to be considered best practice, and most of us even configured .local’s on non-SBS environments. This has never really posed any problems for us I.T. guys, except for a few configuration considerations when setting up Outlook clients, DNS, etc…

Now if you’re like me, another thing I always configured, was user accounts that didn’t match e-mail addresses. An example would be “John Doe”, with the username of “JohnD”, and the e-mail address of “”. Also, our buddy John Doe would have a AD UPN JohnD@internaldomain.local (this was automatically populated on user setup)

User’s Name: John Doe


Username: JohnD

AD UPN: JohnD@internaldomain.local

E-mail Address:


I always liked this as it provided some protection if the users password ever got compromised (in a phishing attack, fake e-mail logon page, etc…), as the password could not actually authenticate when using the e-mail address as a username (the username was never actually provided in the attack, only e-mail).

Now let’s flash forward to this migration from SBS 2008, to Windows Server 2012 R2 with Essentials Experience, and throw Exchange 2013 in to the mix. Right off the bat, everything is working fine, Outlook 2010 is working great, Outlook 2013 is working great. Then BAM, Outlook 2016 comes out!

Outlook 2016 does not allow manual or custom configuration of Exchange accounts. They do this for “reliability” and ease of configuration. This means that you HAVE to have autodiscover setup, and working fluidly. No more manual configuration. Internally inside of the LAN this is all automatic if you configured Exchange properly, but you will have to configure autodiscover externally.

Internally on the LAN, Outlook 2016 clients have absolutely no issues, and authentication is working fine (no password prompts). However, when configuring external users, while you can eventually get it configured, the user is constantly prompted for credentials on every Outlook start.

On these password prompts, you’ll notice it’s authenticating for the users e-mail address. In this example, it’s asking for “” and you enter: “INTERNALDOMAIN\JohnD” and their password, it work for the session, but keeps prompting on every fresh Outlook start.

I did massive amounts of research and seriously I could not come across one article that actually provided all the information I needed, it almost seemed as if this problem was specific to this single environment. Of course, this makes me think I have something configured incorrectly, and I literally spend forever searching for information, checking my VirtualDirectories on my Exchange server, checking logs, wasting tons and tons of time.

Finally after checking my configurations 6-10 times each and spending weeks, I realized it had nothing to do with anything configured incorrectly.

Outlook 2016 does all the configuration automatically, and expects to find everything it needs via auto discover. Putting it simple, the user’s UPN must match their e-mail address.

This means we have to change John Doe’s Active Directory UPN to match his e-mail address. The SAMAccountName still remains the same, so his login to his computer will not change, however after the change he will now be able to log in both with INTERNALDOMAIN\JohnD and

First we have to add the UPN suffix (which is the actual e-mail address domain name) to the Active Directory Domain and Trusts. Instructions are available here:

After adding your e-mail domain to the UPN suffix list. When you go in to “Active Directory Users and Computers”, and view a user’s properties, you’ll notice in the UPN section, you can drop it down and change it from internaldomain.local, to (using my example domains). You can also change the username inside of the UPN.


Essentially for Johny boy, his AD properties window now looks like:

User Logon Name: (we changed the name, and chose the external domain in the drop down to the right)

User logon name (pre-Windows 2000):

INTERNALDOMAIN\ JohnD (we left this the way it was)


John can now login either using “INTERNALDOMAIN\JohnD” or “”. As far as John is concerned we haven’t changed anything and he still logs in using the same format he always has, totally unaware of any changes.

Surprise surprise, autodiscover is now fully functioning for this user. Not only for easy configuration on mobile devices (iPhones, Windows Phones, etc…), but he can now load up Outlook 2016 away from the LAN on the Internet, type in his e-mail address, password, and BAM he’s good to go!

I am a little bit unsettled in the fact that the e-mail address now becomes a fully accepted username on the domain (for security reasons), but I guess we’re stuck with that!


In short, our problem is:

  1. Username doesn’t match e-mail (JohnD username, email)
  2. Running Outlook 2016 and forced to use auto-discover, repeated password prompts
  3. Running .local domain internally, while using different domain externally

In Short, to fix this:

  1. Add UPN Suffix to Active Directory
  2. Change users properties so that UPN matches e-mail address, DO NOT CHANGE the old DOMAIN\Username setting

Other Considerations:

  1. Password prompts on Outlook clients can mean a whole bunch of different problems totally unrelated to this configuration and issue. Always fully diagnose the issue and confirm the issue before applying fixes. Password prompts can mean authentication problems, problems with Exchange’s virtualdirectories, issues with autodiscover, issues with certificate configuration, etc…
  2. If this is your specific issue, you can write a script to run through and update the UPNs on all the accounts. I generally don’t like scripts touching user accounts, so I’m slowly rolling out these changes per user when upgrading them to Outlook 2016. Doing this one by one as we upgrade, allows us to make sure that none of their mobile devices are affected by the UPN change.
  3. Since we are changing UPNs, this could have a major effect on any 3rd party applications that integrate with Active Directory that use UPNs. Always test, and make sure you don’t break any integration points to your 3rd party applications or line of business systems.


Sep 232016

Well, recently one of the servers I monitor and maintain in a remote oil town recently started throwing out a Windows event log warning:

Event ID: 129

Source: HpCISSs2

Description: Reset to device, \Device\RaidPort0, was issued.


The server is an HP ML350p Gen8 (Windows Server 2008 R2) running latest firmware and management software. It has 2 RAID Arrays (RAID1, and RAID5), and a total of 6 disks.

Researching this error, I read that most people had this occur when running the latest HP WBEM providers, as well as anti-virus software. In our case, I actually tried to downgrade to an older version, but noticed the warning still occurs. While we do have anti-virus, it’s not actively scanning (only weekly scheduled scans).

In the process of troubleshooting, I noticed that under the HP Systems Management Homepage, one of the drives in the RAID1 array, had the following stats:

Hard Read Erros:  150
Recovery Read Errors:  7
Total Seeks:  0
Seek Errors:  0

I found these numbers to be very high in my experience. None of the other drives had anything close to this (in 4 years of running, only one other disk had a read error (a single one), this disk however had tons. For some reason the drive is still reporting as operational, when I’d expect it to be marked as a predicted failure, or failed.

While all online documentation was pointing towards at locks on the array by software, from my own experience I think it was actually the array waiting for a read operation on the array, and it was this single disk that was causing a threshold to be hit in the driver, that caused a retry to recover the read operation.

Called up HPe support, I mentioned I’d like to have the drive replaced. The support engineer consulted her senior engineer and reviewed the evidence I presented (along with ADU reports, and Active Monitoring health reports), the senior engineer concurred that the drive should be replaced.

Replacing the drive resolved the issue. I’m also noticing a performance increase on the array as well.

Make sure to always check the stats on the individual components of your RAID arrays, even if everything is operating sound.

Sep 102016

When initiating manual backups or occasionally when automatic/scheduled backups run, a user may notice that Windows Server Backup may appear to “hang” when the status is reporting: “Preparing media to store backups…”.

In some rare cases, it may actually be in a hang state, however most of the time, it’s actually consolidating and/or checking previous backups on the destination media.

To Confirm this:

Open the Task Manager as Administrator, then click on the “Performance” tab, click on “Open Resource Monitor”. Flip over to the “Disk” tab, expand “Disk Activity”, and sort by name. You should see the read requests on the destination media, you’ll also notice that it is slowly progressing consecutively through each backup set (increments of 1, accessing multiple at a time).

This confirms that the Windows Server Backup services are functioning and it is in fact running. In one case, I had 723 previous backups, and it took around 50 minutes to count from 1 to 723, and then the backup finally proceeded.

I have also seen this occur when a previous backup failed or was cancelled. This occurs with Windows Server Backup on Windows Server 2008, Windows Server 2008 R2, and Windows Server 2012 R2.

Sep 082016

If you’re like me, you probably have your Microsoft account configured the same as your e-mail address. While many people use or addresses, some of us prefer to use our actual real e-mail addresses as Microsoft account logins.

Recently, I did a fresh install of Windows 10 on my Microsoft Surface Pro. After joining the Surface to my domain, and attached my Microsoft account, I went to add my Exchange account (which is the same e-mail address I use for my Microsoft account). When trying to add, I was presented with:

There’s already an account set up to use <e-mail address>. (Account Name)

This message stopped me from configuring my Exchange account with the Windows 10 Mail, Calendar, and People apps. Researching this, I noticed numerous other people reporting this problem on multiple forums, however no one had a fix.

It appears there is a conflict with the Microsoft Account (which of course has it’s own mail, calendar, and contacts), and a separate account with the same e-mail address.

To resolve this, I restarted the machine, and logged in using a different account. I then went to “System” under control panel, “Advanced System Settings”, “Advanced” tab, then “Settings” under “User Profiles”. I then proceeded to delete the user profile and restart the system. I confirmed the user profile was fully deleted and then logged back in. Now at this point, the key is to create the Exchange (or any other mail account) before you actually attach your Microsoft account to your system login account. By configuring the e-mail account first, it will avoid this issue.

PLEASE NOTE: By deleting your user profile, you delete all of the contents of the Desktop, My Documents, Music, Pictures, settings, etc… I’d only recommend this if you have either backed up, or are performing this on a fresh install where you currently don’t have any files.

Aug 202016

I’ve decided to setup a Facebook page for my readers to reach out and engage with me about the material and topics I post on my blog.

I’d love to hear your feedback, ideas, and if you’d like to see any types of new content.

The time I’ve wasted on technology

Please feel free to “Like” my Blog’s page on Facebook at:

Aug 202016

I just wanted to create a post about this file. I’m sure some admins have seen this and wondered what it was. The “BitlockerActiveMonitoringLogs” file on the system root directory, present on Microsoft Exchange 2013 servers.

I first noticed this on a clients setup, at first assuming the worst believing the system may have been compromised. However I have seen this file on multiple Exchange installs, on multiple clients, even in my own environment, and can confirm is it present no matter what the CU release level is, thus confirming it has nothing to do with being compromised.

Date modified I’m expecting reflects last system boot-up.

Surprised to see that there are no articles online regarding this file when searching for it specifically, so I decided to create this post to let you know you’re not alone, and the file probably is a system file.