Stephen Wagner

Name: Stephen Wagner

Age: 29

Location: Calgary, Alberta (Canada)

Occupation: President of Digitally Accurate Inc. (also operating as D.A. Consulting)


-Computers (Windows, Linux, OSX)

-Wireless Technologies (Device hacking, reverse engineering, long range links, open source hardware)

-Single Board Computers (SBCs, SBC Development)

-Mobile Platforms (Windows 10 for Mobile)

-Mountain Biking

-Electronica (House, Hard House)




-Disaster Recovery

-HP Hardware (Proliant Servers, SANs)

-Storage (iSCSI, SAN, NFS, NAS)


Microsoft Lumia 950XL (Dual SIM) (Insider Fast Build Flight)

Nokia Lumia 1020 (Insider Fast Build Flight) (Backup Phone)

Microsoft Surface Pro

Lenovo X1 Carbon 2015

Mini Biography:

Born and raised in Calgary, Alberta (3rd generation), I’ve had the pleasure of also living in North Vancouver and Maple Ridge in British Columbia, before moving back home to Calgary.

I started out very young with technology. My father owned numerous Computer/I.T. companies over the years doing everything from business systems, to Point of Sale solutions. Growing up I had access to business hardware, software, and pretty much all the fun stuff that comes with it.

First memories I have go back to the days of Windows for Workgroup 3.11, NT 3.51, and Unixware around the age of 7. It was around this time that I really started my passion with computers and technology. I still remember how amazed I was when the NT 4.0 beta was released. I was fully building computers at this age (and my father was getting me to help with building systems and rolling them out to clients).

My main obsession was with networking. I found the premise of connecting computers to a large network and sharing information very fascinating, whether it was talking and chatting, sharing files and data, or playing multiplayer games. Around the age of 10 I was already wrapping IPX/SPX inside of TCP to play LAN games over the internet against other players, and had developed a thorough understanding of network technologies.

Shortly after this, my fascination with Linux started. I picked up my first copy of RedHat Linux 5.1 when I was 11 years old from a book purchased at a bookstore. And so began my dive in to the Linux world (I can’t tell you how many nights were spent doing live FTP installs over dial-up connections to get the latest versions of Linux).

Over the years I learnt about domains, active directory, clustering, and got more interested in business applications, infrastructure, and internet services.

Age 13 through 17 were spent learning about internet services, wireless services (and wireless hacking), bettering my knowledge of Linux, and more business infrastructure services. At 16 I could fully configure a Windows 2000 Advanced Server with all of the add-ons/roles enabled and configured (AD, Exchange, FTP, Telnet, Remote Installation services, DHCP, DNS, Clustering, etc…).

Around 17 years old I really started to dive in to Single Board Computing (using alternative architectures such as ARM and MIPS), and wireless technologies. I started off with a Soekris Net4801 board, doing remote PXE installations of RedHat Linux using PXE and NFS. I also picked up specialize wireless cards and specialized wireless antennas (learning about promiscuous mode on wireless cards, long range wireless links, and customization of wireless technologies to permit long range links). Over the years I also made it a hobby of installing/hacking Linux on everything I could (proprietary firewalls, Xbox, Wireless Access Points, etc…). I can’t verify this, but I think I was one of the first people to get the Redhat 9 distribution running on the First Generation Xbox.

Upon graduating high school in 2004, I initially was pre-accepted in to Electrical Engineering, however decided against immediately going to university so that I could get a job and save up beforehand.

My first big jump in to business was in 2004 as I was brought onboard with a major homebuilder as an I.T. Specialist.

During my time there, within 2 years I was promoted numerous times first to I.T. Coordinator, and finally to I.T. Manager. I managed over 80 workstations deployed across two offices (in Calgary and Edmonton), and over 20 sales centers split between the two cities. The technologies I implemented, managed, and supported included Windows Server 2003, Citrix, Terminal Services, SQL Server, and a number of Line of Business applications specific to the industry.

During this time, I fell in love with business and business I.T. infrastructure. After 2 years I resigned from my position to start my own business (at the age of 19).

For over 10 years I have provided I.T. Infrastructure services for the SMB and enterprise markets, specializing in Virtualization, Line of Business applications, ERP/CRM solutions, Storage, Security, and I.T. Management.

Experienced with:
Infrastructure design, implementation and support
Virtualization (vSphere 5.x, vSphere 6.x, iSCSI, HP MSA 2040 SAN)
Disaster Recovery (Tape Libraries, Disk Backup, special requirement backups for databases, etc...)
Wireless Point to Point (Linking buildings)
Wireless Point to Multi-Point (Warehouse wireless)
Linking multi-sites using fiber
Line of Business Implementation and Integration (Accounting, Estimating, Purchasing, Scheduling, etc...)
Remote Access Solutions (Linking multiple branch offices to HQ, or remote access for employees from home)
Mobile Devices (Windows Phone, Windows Mobile, Apple iPhone, BlackBerry, Blackberry Enterprise Server)
Workstation Management Technologies
IT Workflow and IT Support Development/Management
Line of Business Applications (Directional Drilling, Home Sales, Shop Management)

Software Implementation, Integration, and Support Experience:
Microsoft Windows Server 2000, 2003, 2008, 2008R2, 2012, 2012 R2
Microsoft Windows Small Business Server (SBS2008, SBS2011)
Microsoft Exchange Server (2007, 2010, 2013, 2016)
Microsoft SQL Server (2005, 2008, 2008R2)
RedHat Enterprise Linux
Symantec Endpoint Protection (and Protection Suite)
Sophos Unified Threat Management (Sophos UTM)
Simply Accounting
Sage BusinessVision
Timberline Accounting and Estimating
BuilderMT Construction Management
Zybertech HomeFront
Lio-Target iSCSI
Embedded Linux development
Halliburton Landmark Compass, WellPlan, EDM
Shoptech E2

Get in Touch:


Dec 082016

So you just completed your migration from an earlier version of vSphere up to vSphere 6.5 (particularly vCenter 6.5 Virtual Appliance). When trying to log in to the vSphere web client, you receive numerous “The VMware enhanced authentication plugin has updated it’s SSL certificate in Firefox. Please restart Firefox.”. You’ll usually see 2 of these messages in a row on each page load.

You’ll also note that the “Enhanced Authentication Plugin” doesn’t function after the install (it won’t pull your Active Directory authentication information).

To resolve this:

Uninstall all vSphere plugins from your workstation. I went ahead and uninstalled all vSphere related software on my workstation, this includes the deprecated vSphere C# client application, all authentication plugins, etc… These are all old.

Open up your web browser and point to your vCenter server (https://vCENTERSERVERNAME), and download the “Trusted root CA certificates” from VMCA (VMware certificate authority).

Download and extract the ZIP file. Navigate through the extracted contents to the windows certs. These root CA certificates need to be installed to your “Trusted Root Certification Authorities” store on your system, and make sure you skip the “Certificate Revocation List” file which ends in a “.r0”.

To install them, right click, choose “Install Certificate”, choose “Local Machine”, yes to UAC prompt, then choose “Place all certificates in the following store”, browse, and select “Trusted Root Certification Authorities”, and finally finish. Repeat for each of the certificates. Your workstation will now “trust” all certificates issued by your VMware Certificate Authority (VMCA).

You can now re-open your web browser, download the “Enhanced Authentication Plugin” from your vCenter instance, and install. After restarting your computer, the plugin should function and the messages will no longer appear.

Leave a comment!

Dec 072016

Well, I start writing this post minutes after completing my first vSphere 6.0 upgrade to vSphere 6.5, and as always with VMware products it went extremely smooth (although with any upgrade there are minor hiccups).

Thankfully with the evolution of virtualization technology, upgrades such as the upgrade to vSphere 6.5 is such a massive change to your infrastructure, yet the process is extremely simplified, can be easily rolled out, and in the event of problems has very simple clear paths to revert back and re-attempt. Failed upgrades usually aren’t catastrophic, and don’t even affect production environments.

Whenever I do these vSphere upgrades, I find it funny how you’re making such massive changes to your infrastructure with each click and step, yet the thought process and understanding behind it is so simple and easy to follow. Essentially, after one of these upgrades you look back and think: “Wow, for the little amount of work I did, I sure did accomplish a lot”. It’s just one of the beauties of virtualization, especially holding true with VMware products.

To top it all off you can complete the entire upgrade/migration without even powering off any of your virtual machines. You could do this live, during business hours, in a production environment… How cool is that!


Just to provide some insights in to my environment, here’s a list of the hardware and configuration:

-2 X HPe Proliant DL360p Gen8 Servers (each with dual processors, and each with 128GB RAM, no local storage)

-1 X HPe MSA2040 Dual Controller SAN (each host has multiple connections to the SAN via 10Gb DAC iSCSI, 1 connection to each of the dual controllers)

-VMware vSphere 6.0 running on Windows Virtual Machine (Windows Server 2008 R2)

-VMware Update Manager (Running on the same server as the vCenter Server)

-VMware Data Protection (2 x VMware vDP Appliances, one as a backup server, one as a replication target)

-VMware ESXi 6.0 installed on to SD-cards in the servers (using HPe Customized ESXi installation)


One of the main reasons why I was so quick to adopt and migrate to vSphere 6.5, was I was extremely interested in the prospect of migrating a Windows based vCenter instance, to the new vCenter 6.5 appliance. This is handy as it simplifies the environment, reduces licensing costs and requirements, and reduces time/effort on server administration and maintenance.

First and foremost, following the recommended upgrade path (you have to specifically do the upgrades and migrations for all the separate modules/systems in a certain order), I had to upgrade my vDP appliances first. For vDP to support vCenter 6.5, you must upgrade your vDP appliances to 6.1.3. As with all vDP upgrades, you must shut down the appliance, mark all the data disks as dependent, take a snapshot, and mount the upgrade ISO, and then boot and initiate the upgrade from the appliance web interface. After you complete the upgrade and confirm the appliance is functioning, you shut down the appliance, remove the snapshot, mark all data disks as independent (except the first Virtual disk, you only mark virtual disk 2+ and up as independent), and you’re done your upgrade.

A note on a problem I dealt with during the upgrade process for vDP to version 6.1.3 (appliance does not detect mounted ISO image) can be found here:


Moving on to vCenter! VMware did a great job with this. You load up the VMware Migration Assistant tool on your source vCenter server, load up the migration/installation application on a separate computer (the workstation you’re using), and it does the rest. After prepping the destination vCenter appliance, it exports the data from the source server, copies it to the destination server, shuts down the source VM, and then imports the data to the destination appliance and takes over the role. It’s the coolest thing ever watching this happen live. Upon restart, you’ve completed your vCenter Server migration.

A note on a problem I dealt with during the migration process (which involved exporting VMware Update Manager from the source server) can be found here:


And as for the final step, it’s now time to upgrade your ESXi hosts to version 6.5. As always, this is an easy task with VMware Update Manager, and can be easily and quickly rolled out to multiple ESXi hosts (thanks to vMotion and DRS). After downloading your ESXi installation ISO (in my case I use the HPe customized image), you upload it in to your new VMware Update Manager instance, add it to an upgrade baseline, and then attach the baseline to your hosts. To push this upgrade out, simply select the cluster or specific host (depending on if you want to rollout to a single host, or multiple at once), and remediate! After a couple restarts the upgrade is done.

A note on a problem I dealt with during ESXi 6.5 upgrade (conflicting VIBs marking image as incompatible when deploying HPe customized image) can be found here:


After all of the above, the entire environment is now running on vSphere 6.5! Don’t forget to take a backup before and after the upgrade, and also upgrade your VM hardware versions to 6.5 (VM compatibility version), and upgrade VMware tools on all your VMs.

Make sure to visit https://YOURVCENTERSERVER to download the VMware Certificate Authority (VMCA) root certificates, and add them to the “Trusted Root Certification Authorities” on your workstation so you can validate all the SSL certs that vCenter uses. Also, note that the vSphere C# client (the windows application) has been deprecated, and you now must use the vSphere Web Client, or the new HTML5 web client.

Happy Virtualizing! Leave a comment!

Dec 072016

After successfully completing the migration from vCenter 6.0 (on Windows) to the vCenter 6.5 Appliance, all I had remaining was to upgrade my ESXi hosts to ESXi 6.5.

In my test environment, I run 2 x HPe Proliant DL360p Gen8 servers. I also have always used the HPe customized ESXi image for installs and upgrades.

It was easy enough to download the customized HPe installation image from VMware’s website, I then loaded it in to VMware Update Manager on the vCenter appliance, created a baseline, and was prepared to upgrade the hosts.

I successfully upgraded one of my hosts without any issues, however after scanning on my second host, it reported the upgrade as incompatible and stated: “The upgrade contains the following set of conflicting VIBs: Mellanox_bootbank_net.XXXXversionnumbersXXXX. Remove the conflicting VIBs or use Image Builder to create a custom ISO.”

I checked the host to see if I was even using the Mellanox drivers, and thankfully I wasn’t and could safely remove them. If you are using the drivers that are causing the conflict, DO NOT REMOVE them as it could disconnect all network interfaces from your host. In my case, since they were not being used, uninstalling them would not effect the system.

I SSH’ed in to the host and ran the following commands:

esxcli software vib list | grep Mell (This shows the VIB package that the Mellanox driver is inside of. In my case, it returned “net-mst”)

esxcli network nic list (this command verifies which drivers you are using on your network interfaces on the host)

esxcli software vib remove -n net-mst (this command removes the VIB that contains the problematic driver)

After doing this, I restarted the host, scanned for upgrades, and successfully applied the new vCenter 6.5 ESXi Customized HPe image.

Leave a comment!

Dec 072016

During my first migration from VMware vCenter 6.0 to VMware vCenter 6.5 Virtual appliance, the migration failed. The migration installation UI would shutdown the source VM, and numerous errors would occur afterwards when the destination vCenter appliance would try finishing configuration.

If you were monitoring the source vCenter server, during the export process, one would notice that an error pops up while compressing the source data. The error presented is generated from Windows creating an archive (zip file), the error reads: “The compressed (zipped) folder is invalid or corrupted.”. The entire migration process halts until you dismiss this message, with the entire migration ultimately failing (at first it appears to continue, but ultimately fails).

If you continued, and had the migration fail. You’ll need to power off the failed (new) vCenter appliance (it’s garbage now), and you’ll need to power on the source (original) vCenter server. The active directory trust will no longer exist at this point, so you’ll need to log on with a local (non-domain) account (on the source server), and re-create the computer trust on the domain using the netdom command:


After re-creating the trust, restart the original vCenter server. You have now reverted to your original vCenter instance and can retry the migration.

Now back to the main issue. I tried a bunch of different things and wasted an entire evening (checking character lengths on paths/filenames, trying different settings, pausing processes in case timeouts were being hit, etc…) however finally I noticed that the compression archive would crash/fail on a file called “vum_registry”.

VUM brings VMware Update Manager to mind, which I do have installed, configured, and running.

I went ahead and uninstalled VMware Update Manager off my source server (as it’s easy enough to re-configure from scratch after the migration). I then proceeded to initiate a migration. To my surprise, the “data to migrate” went from 7.9GB to 2.4GB. This is a huge sign that something was messed up with my VMware update manager deployment (even though it was working fine). I’m assuming there were either filenames that were too long (exceeded the 260 character limit on paths and filenames), special characters were being used where they shouldn’t, or something else was messed up.

After the uninstall of Update Manager, the migration completed successfully. Leave a comment!

Dec 052016

In the process of prepping my test environment so I can upgrade from vSphere 6.1 to 6.5, one of the prerequisites is to first upgrade your VDP appliances to version 6.1.3 (6.1.3 is the only version of VDP that supports vSphere 6.5). In my environment I’ll be upgrading VDP from 6.1.2 to 6.1.3.

After downloading the ISO, changing my disks to dependant, creating a snapshot, and attaching the ISO to the VM. My VDP appliances would not recognize the ISO image, showing the dreaded: “To upgrade your VDP appliance, place connect a valid upgrade ISO image to the appliance.”.


I tried a few things, including trying the old “patch” that was issues for 6.1 when it couldn’t detect, unfortunately it didn’t help. I also tried to manually mount the virtual CD-Rom to the mountpoint but had no luck. The mountpoint /mnt/auto/cdrom is locked by the autofs service. If you try to modify these files (such as delete, create, etc…), you’ll encounter a bunch of errors and have no luck (permission denied, file and/or directory doesn’t exist, etc…).

Essentially the autofs service was not auto-mounting the virtual CD drive to the mount point.

To fix this:

  1. SSH in to the VDP appliance
  2. Run command “sudo su” to run commands as root
  3. Use vi to edit the auto.mnt file using command: “vi /etc/auto.mnt”
  4. At the end of the first line in the file, you will see “/dev/cdrom” (without quotation), change this to “/dev/sr0” (again, without quotation)
  5. Save the file (after editing the text, Ctrl+c, then type “:w” and enter which writes the file, then type “:q” then enter to quit vi.
  6. Reload the autofs config using command: “/etc/init.d/autofs reload”
  7. At the shell, run “mount” to show the active mountpoints, you’ll notice the ISO is now mounted after a few seconds.
  8. You can now initiate the upgrade. Start it.
  9. At 71%, autofs updates via a RPM, and the changes you made to the config are cleared. IMMEDIATELY edit the /etc/auto.mnt file again, change “/dev/cdrom” to “/dev/sr0” and save the file, and issue the command “/etc/init.d/autofs reload”. Do this as fast as possible.
  10. You’re good to go, the install will continue and take some time. The web interface will fail, and become unresponsive. Simply wait, and the vDP appliance will eventually shut down (in my case it took over 30 minutes after the web interface failed to reconnect, in a high performance environment for the vDP VM to shut down).

And done! Leave a comment!


Nov 102016

I recently got my hands on a D-Link DCS-5222L IP Camera and thought it would be pretty cool to get it setup to do YouTube Live streaming. To configure the stream I would have to find a way to convert and encode the RTSP stream from the camera to a YouTube Live compatible RTMP stream.

I figured I would see if I could use VLC or FFMPEG to convert the stream. I had no luck with VLC however I found out ffmpeg should do what I needed to do. I couldn’t easily find the proper usage command and flags, however eventually after viewing multiple sites, mixing, matching, and testing, I found this command worked:

ffmpeg -rtsp_transport tcp -i rtsp://IP.CAM.ADD.RESS/live1.sdp -framerate 30 -video_size 1280×720 -vcodec libx264 -preset veryfast -maxrate 1984k -bufsize 3968k -vf “format=yuv420p” -g 60 -c:a aac -b:a 128k -ar 44100 -f flv rtmp://YOUTUBELIVE/DIR/StreamKey

Happy streaming! And if anyone can refine, or recommend better command usage, please post in the comments!

Nov 052016

Yesterday, I had a reader (Nicolas) leave a comment on one of my previous blog posts bringing my attention to the MTU for Jumbo Frames on the HPe MSA 2040 SAN.

MSA 2040 MTU Comment









Since I first started working with the MSA 2040. Looking at numerous HPe documents outlining configuration and best practices, the documents did confirm that the unit supported Jumbo Frames. However, the documentation on the MTU was never clearly stated and can be confusing. I was under the assumption that the unit supported 9000 MTU, while reserving 100 bytes for overhead. This is not necessarily the case.

Nicolas chimed in and provided details on his tests which confirmed the HPe MSA 2040 does actually have a working MTU of 8900. In my configuration I did the tests (that Nicolas outlined), and confirmed that the MTU would cause packet fragmentation if the MTU was greater than 8900.

ESXi vmkping usage:

This is a big discovery because packet fragmentation will not only degrade performance, but flood the links with lots of packet fragmentation.

I went ahead and re-configured my ESXi hosts to use an MTU of 8900 on the network used with my SAN. This immediately created a MASSIVE performance increase (both speed, and IOPS). I highly recommend that users of the MSA 2040 SAN confirm this on their own, and update the MTUs as they see fit.

Also, this brings up another consideration. Ideally, on a single network, you want all devices to be running the same MTU. If your MSA 2040 SAN is on a storage network with other SAN devices (or any other device), you may want to configure all of them to use the MTU of 8900 if possible (and of course, don’t forget your servers).

A big thank you to Nicolas for pointing this out!

Sep 232016

There’s quite a few of us that started off deploying Small Business Server (SBS2008, SBS2011) environments back in the day, loving the handy all-in-one package taking care of everything from Active Directory and Exchange, to disaster recovery and business continuity. However, some of these old environments are starting to catch up with us. I wanted to open a discussion on a big issue I had a couple years ago in one of my first migrations from SBS 2008, to Windows Server 2012 R2 with the Essentials Experience role installed, with Exchange Server 2013.

As most of you know, SBS comes packaged to push “.local” domains on initial domain configuration. This used to be considered best practice, and most of us even configured .local’s on non-SBS environments. This has never really posed any problems for us I.T. guys, except for a few configuration considerations when setting up Outlook clients, DNS, etc…

Now if you’re like me, another thing I always configured, was user accounts that didn’t match e-mail addresses. An example would be “John Doe”, with the username of “JohnD”, and the e-mail address of “”. Also, our buddy John Doe would have a AD UPN JohnD@internaldomain.local (this was automatically populated on user setup)

User’s Name: John Doe


Username: JohnD

AD UPN: JohnD@internaldomain.local

E-mail Address:


I always liked this as it provided some protection if the users password ever got compromised (in a phishing attack, fake e-mail logon page, etc…), as the password could not actually authenticate when using the e-mail address as a username (the username was never actually provided in the attack, only e-mail).

Now let’s flash forward to this migration from SBS 2008, to Windows Server 2012 R2 with Essentials Experience, and throw Exchange 2013 in to the mix. Right off the bat, everything is working fine, Outlook 2010 is working great, Outlook 2013 is working great. Then BAM, Outlook 2016 comes out!

Outlook 2016 does not allow manual or custom configuration of Exchange accounts. They do this for “reliability” and ease of configuration. This means that you HAVE to have autodiscover setup, and working fluidly. No more manual configuration. Internally inside of the LAN this is all automatic if you configured Exchange properly, but you will have to configure autodiscover externally.

Internally on the LAN, Outlook 2016 clients have absolutely no issues, and authentication is working fine (no password prompts). However, when configuring external users, while you can eventually get it configured, the user is constantly prompted for credentials on every Outlook start.

On these password prompts, you’ll notice it’s authenticating for the users e-mail address. In this example, it’s asking for “” and you enter: “INTERNALDOMAIN\JohnD” and their password, it work for the session, but keeps prompting on every fresh Outlook start.

I did massive amounts of research and seriously I could not come across one article that actually provided all the information I needed, it almost seemed as if this problem was specific to this single environment. Of course, this makes me think I have something configured incorrectly, and I literally spend forever searching for information, checking my VirtualDirectories on my Exchange server, checking logs, wasting tons and tons of time.

Finally after checking my configurations 6-10 times each and spending weeks, I realized it had nothing to do with anything configured incorrectly.

Outlook 2016 does all the configuration automatically, and expects to find everything it needs via auto discover. Putting it simple, the user’s UPN must match their e-mail address.

This means we have to change John Doe’s Active Directory UPN to match his e-mail address. The SAMAccountName still remains the same, so his login to his computer will not change, however after the change he will now be able to log in both with INTERNALDOMAIN\JohnD and

First we have to add the UPN suffix (which is the actual e-mail address domain name) to the Active Directory Domain and Trusts. Instructions are available here:

After adding your e-mail domain to the UPN suffix list. When you go in to “Active Directory Users and Computers”, and view a user’s properties, you’ll notice in the UPN section, you can drop it down and change it from internaldomain.local, to (using my example domains). You can also change the username inside of the UPN.


Essentially for Johny boy, his AD properties window now looks like:

User Logon Name: (we changed the name, and chose the external domain in the drop down to the right)

User logon name (pre-Windows 2000):

INTERNALDOMAIN\ JohnD (we left this the way it was)


John can now login either using “INTERNALDOMAIN\JohnD” or “”. As far as John is concerned we haven’t changed anything and he still logs in using the same format he always has, totally unaware of any changes.

Surprise surprise, autodiscover is now fully functioning for this user. Not only for easy configuration on mobile devices (iPhones, Windows Phones, etc…), but he can now load up Outlook 2016 away from the LAN on the Internet, type in his e-mail address, password, and BAM he’s good to go!

I am a little bit unsettled in the fact that the e-mail address now becomes a fully accepted username on the domain (for security reasons), but I guess we’re stuck with that!


In short, our problem is:

  1. Username doesn’t match e-mail (JohnD username, email)
  2. Running Outlook 2016 and forced to use auto-discover, repeated password prompts
  3. Running .local domain internally, while using different domain externally

In Short, to fix this:

  1. Add UPN Suffix to Active Directory
  2. Change users properties so that UPN matches e-mail address, DO NOT CHANGE the old DOMAIN\Username setting

Other Considerations:

  1. Password prompts on Outlook clients can mean a whole bunch of different problems totally unrelated to this configuration and issue. Always fully diagnose the issue and confirm the issue before applying fixes. Password prompts can mean authentication problems, problems with Exchange’s virtualdirectories, issues with autodiscover, issues with certificate configuration, etc…
  2. If this is your specific issue, you can write a script to run through and update the UPNs on all the accounts. I generally don’t like scripts touching user accounts, so I’m slowly rolling out these changes per user when upgrading them to Outlook 2016. Doing this one by one as we upgrade, allows us to make sure that none of their mobile devices are affected by the UPN change.
  3. Since we are changing UPNs, this could have a major effect on any 3rd party applications that integrate with Active Directory that use UPNs. Always test, and make sure you don’t break any integration points to your 3rd party applications or line of business systems.


Sep 232016

Well, recently one of the servers I monitor and maintain in a remote oil town recently started throwing out a Windows event log warning:

Event ID: 129

Source: HpCISSs2

Description: Reset to device, \Device\RaidPort0, was issued.


The server is an HP ML350p Gen8 (Windows Server 2008 R2) running latest firmware and management software. It has 2 RAID Arrays (RAID1, and RAID5), and a total of 6 disks.

Researching this error, I read that most people had this occur when running the latest HP WBEM providers, as well as anti-virus software. In our case, I actually tried to downgrade to an older version, but noticed the warning still occurs. While we do have anti-virus, it’s not actively scanning (only weekly scheduled scans).

In the process of troubleshooting, I noticed that under the HP Systems Management Homepage, one of the drives in the RAID1 array, had the following stats:

Hard Read Erros:  150
Recovery Read Errors:  7
Total Seeks:  0
Seek Errors:  0

I found these numbers to be very high in my experience. None of the other drives had anything close to this (in 4 years of running, only one other disk had a read error (a single one), this disk however had tons. For some reason the drive is still reporting as operational, when I’d expect it to be marked as a predicted failure, or failed.

While all online documentation was pointing towards at locks on the array by software, from my own experience I think it was actually the array waiting for a read operation on the array, and it was this single disk that was causing a threshold to be hit in the driver, that caused a retry to recover the read operation.

Called up HPe support, I mentioned I’d like to have the drive replaced. The support engineer consulted her senior engineer and reviewed the evidence I presented (along with ADU reports, and Active Monitoring health reports), the senior engineer concurred that the drive should be replaced.

Replacing the drive resolved the issue. I’m also noticing a performance increase on the array as well.

Make sure to always check the stats on the individual components of your RAID arrays, even if everything is operating sound.

Sep 102016

When initiating manual backups or occasionally when automatic/scheduled backups run, a user may notice that Windows Server Backup may appear to “hang” when the status is reporting: “Preparing media to store backups…”.

In some rare cases, it may actually be in a hang state, however most of the time, it’s actually consolidating and/or checking previous backups on the destination media.

To Confirm this:

Open the Task Manager as Administrator, then click on the “Performance” tab, click on “Open Resource Monitor”. Flip over to the “Disk” tab, expand “Disk Activity”, and sort by name. You should see the read requests on the destination media, you’ll also notice that it is slowly progressing consecutively through each backup set (increments of 1, accessing multiple at a time).

This confirms that the Windows Server Backup services are functioning and it is in fact running. In one case, I had 723 previous backups, and it took around 50 minutes to count from 1 to 723, and then the backup finally proceeded.

I have also seen this occur when a previous backup failed or was cancelled. This occurs with Windows Server Backup on Windows Server 2008, Windows Server 2008 R2, and Windows Server 2012 R2.