VMware Archives - The time I've wasted on technology...

VMware ESXi Boot Failure from USB or SD Card with HPe iLO Amplifier and InfoSight

Feb 182019

Unable to boot ESXi from USB or SD Card on HPe Proliant Server

After installing HPe iLO Amplifier on your network and updating iLO 4 firmware to 2.60 or 2.61, you may notice that your HPe Proliant Servers may fail to boot ESXi from a USB drive or SD-Card.

This was occuring on 2 ESXi Hosts. Both were HPe Proliant DL360p Gen8 Servers. One server was using an internal USB drive for ESXi, while the other was using an HPe branded SD Card.

The issue started occuring on both hosts after a planned InfoSight implementation. Both hosts iLO controllers firmware were upgraded to 2.61, iLO Amplifier was deployed (and the servers added), and the amplifier was connected to an HPe InfoSight account.

Please see below for errors:

Errors

Error loading /s.v00
 Compressed MD5: 00000000000000000000000000
 Decompressed MD5: 00000000000000000000000000
 Fatal error: 8 (Device Error)

Error mboot.c32 attempted DOS system call

mboot.c32: attempted DOS system call INT 21 0d00 E8004391
 boot:

Symptoms

This issue may occur intermittently, on the majority of boots, or on all boots. Re-installing ESXi on the media, as well as replacing the USB/SD Card has no effect. Installation will be successful, however you the issue is still experiences on boot.

HPe technical support was unable to determine the root of the issue. We found the source of the issue and reported it to HPe technical support and are waiting for an update.

The Issue and Fix

This issue occurs because the HPe iLO Amplifier is running continuous server inventory scans while the hosts are booting. When one inventory completes, it restarts another scan.

The following can be noted:

iLO Amplifier inventory percentage resets back to 0% and starts again numerous times during the server boot
Inventory scan completes, only to restart again numerous times during the server boot
Inventory scan resets back to 0% during numerous different phases of BIOS initialization and POST.

We noticed that once the HPe iLO Amplifier Virtual Machine was powered off, not only did the servers boot faster, but they also booted 100% succesfully each time. Powering on the iLO Amplifier would cause the ESXi hosts to fail to boot once again.

I’d also like to note that on the host using the SD-Card, the failed boot would actually completely lock up iLO, and would require physical intervention to disconnect and reconnect the power to the server. We were unable to restart the server once it froze (this did not happen to the host using the USB drive).

There are some settings on the HPe iLO amplifier to control performance and intervals of inventory scans, however we noticed that modifying these settings did not alter or stop the issue, and had no effect.

Make sure your iLO amplifier is powered off during any maintenance to avoid hosts freezing/failing to boot.

VMFS Auto UNMAP (Space Reclamation) not working on vSphere

VMware, vSphere No Responses »

Nov 172018

When running VMware vSphere 6 (and later) and ESXi on your hosts with VMFS6, you may notice that auto unmap (space reclamation) is not working even though it is enabled. In addition, you’ll find that manual unmap functions still work.

Why

This is because your storage array (SAN) may have a larger unmap granularity block size than 1MB. VMFS version 6 (source) requires an unmap granularity of 1MB, and does not support automatic unmap on arrays that are larger.

For example, on the HPe MSA 2040 the page size when using virtual storage is 4MB, hence auto unmap is not supported and does not work. You can still perform automatic unmap functions on arrays with block/page sizes larger than 1MB.

Additional Information and Resources

Perform manual VMFS unmap on vSphere 6.5 and 6.7 with VMFS 6 – https://www.stephenwagner.com/2017/02/07/vmfs-unmap-command-vsphere-6-5-with-vmfs-6-auto/

vSphere 6.5, 6.7 and VMFS 6 – Change storage reclaim priority from low to medium or high – https://www.stephenwagner.com/2017/02/08/vsphere-vmfs-6-change-storage-reclaim-priority-low-medium-high/

Release unused space on host and guest filesystems with thin-provisioned Sophos UTM appliance (SW) – https://www.stephenwagner.com/2018/01/18/release-unused-space-vmdk-thin-provisioned-sophos-utm/

VMware vDP (vSphere Data Protection) actively being maintainted for rest of VMware vSphere 6.5 Updates

vDP, VMware, vSphere No Responses »

Oct 212018

VMware announced the end of availability for the vDP (vSphere Data Protection) product last year. In their release, it was mentioned that no more updates would be issued for the product.

At that time, I recommended to most customers to start planning to upgrade/migrate to a different product. VMware then released some ESXi and vSphere upgrades, but didn’t immediately update the vDP appliance. It appeared as though no more updates would be issued.

Due to the the previous announcement, and a lack of an updated vDP appliance, this increased the urgency of migrating to a different product.

However, just recently I noticed that VMware is still maintaining the vDP appliance for vSphere 6.5, so you can continue to use it while you plan your migration to a new Backup/DR product, and your upgrade to vSphere version 6.7.

Please note: I noticed the updates to the vDP appliance are being released later than the vSphere/ESXi updates. I wouldn’t upgrade your ESXi hosts to 6.5 certain update levels until the applicable vDP updated appliance is available for that version.

VMware ESXi asyncUnmapFile grows massive, cannot delete, errors

ESXi, iSCSI, VMware, vSphere No Responses »

Oct 202018

On an ESXi host when performing a manual unmap on your storage datastore, you may notice a very large (hidden) file on the datastore root called “.asyncUnmapFile”. This file could be taking up terabytes of space, and you aren’t able to delete this file.

asyncUnmapFile

Typically the asyncUnmapFile is used by the UNMAP feature on ESXi hosts to deal with unmapping and unallocating storage blocks on the SAN. When you run a manual UNMAP, this file should be created and should appear to using “0” (no) space (even if it is). When an UNMAP completes, this file should disappear and be automatically removed by the function. If an UNMAP is interupted, this file will not be deleted, allowing you to restart the process and upon a full successful completion, it should then be deleted.

The Problem

Some time ago, I had an issue when performing a manual UNMAP, where the ESXi host became unresponsive (due to memory issues). The command appeared to be completed, however I believe it caused potential issues or corruption on the iSCSI datastore. In subsequent runs, the UNMAP appeared to be functioning and working, however I didn’t realize that the asyncUnmapFile had grown to around 1.5TB.

This was noticed during a SAN storage audit, where we saw that the virtual pool on the SAN was using up way more storage than it should be on the datastore.

When we identified the file was this large and causing issues, we attempted to perform 2 UNMAPs (different reclaim sizes) to see if it would be automatically cleared afterwards. It had no effect and the file was unchanged.

We also tried to modify the permissions on the file, however when trying to delete it, it would report that the file or folder was not found, or that it does not exist. This was concerning as we were worried about potential datastore corruption.

It was also noticed that in the hostd.log and vmkernel.log we saw some errors where the host believed that the blocks on the datastore had already been freed: “on volume labeled ‘iSCSIDatastore01’ already freed by another host: This may be a non-issue”

The Solution

Unfortunately with all the research we did, we couldn’t find a clear-cut solution. With worries that the datastore may be corrupted, we needed to do something.

A decision was ultimately made to Storage vMotion all the VMs (Virtual Machines) to another datastore on a separate storage pool, delete the now empty LUN, and recreate it from scratch. After this, we used Storage vMotion again to move the VMs back.

Instantly I noticed that the VMs on that datastore were running faster (it’s only been 12 hours, so I’ll be adding an update in a few days to confirm). We no longer have the file on any of our datastores.

If anyone has further insight in to this issue, please leave a comment!

Procedure for restoring a VM from SAN Storage LUN snapshot

ESXi, iSCSI, MSA, VMware, vSphere No Responses »

Aug 272018

So, what happens in a worst-case scenario where your backup system fails, you don’t have any VM snapshots, and the last thing standing in the way of complete data loss is your SAN storage systems LUN snapshots?

Well, first you fire whoever purchased and implemented the backup system, then secondly you need to start restoring the VM (or VMs) from your SAN LUN snapshots.

While I’ve never had to do this in the past (all the disaster recovery solutions I’ve designed and sold have been tested and function), I’ve always been curious what the process is and would be like. Today I decided to try it out and develop a procedure for restoring a VM from SAN Storage LUN snapshot.

For this test I pretended a VM was corrupt on my VMware vSphere cluster and then restored it to a previous state from a LUN snapshot on my HPe MSA 2040 (identical for the HPe MSA 2050, and MSA 2052) Dual Controller SAN.

To accomplish the restore, we’ll need to create a host mapping on the SAN for the LUN snapshot to a new LUN number available to the hosts. We then need to add and mount the VMFS volume (residing on the snapshot) to the host(s) while assigning it a new signature and then vMotion the VM from the snapshot’s VMFS to original datastore.

Important Notes (Read first):

When mounting a VMFS volume from a SAN snapshot, you MUST RE-SIGNATURE THE SNAPSHOT VMFS volume. Not doing so can cause problems.
The snapshot cannot be mapped as read only, VMFS volumes must be marked as writable in order to be mounted on ESXi hosts.
You must follow the proper procedure to gracefully dismount and detach the VMFS volume and storage device before removing the snapshot’s host mapping on the SAN.
We use Storage vMotion to perform a high-speed move and recovery of the VM. If you’re not licensed for Storage vMotion, you can use the datastore file browser and copy/move from the snapshot VMFS volume to live production VMFS volume, however this may be slower.
During this entire process you do not touch, modify, or change any settings on your existing active production LUNs (or LUN numbers).
Restoring a VM from a SAN LUN snapshot will restore a crash consistent copy of the VM. The VM when recovered will believe a system crash occurred and power was lost. This is NOT a graceful application consistent backup and restore.
Please read your SAN documentation for the procedure to access SAN snapshots, and create host mappings. With the MSA 2040 I can do this live during production, however your SAN may be different and your hosts may need to be powered off and disconnected while SAN configuration changes are made.
Pro tip: You can also power on and initialize the VM from the snapshot before initiating the storage vMotion. This will allow you to get production services back online while you’re moving the VM from the snapshot to production VMFS volumes.
I’m not responsible if you damage, corrupt, or cause any damage or issues to your environment if you follow these procedures.

We are assuming that you have already either deleted the damaged VM, or removed it from your inventory and renamed the VMs folder on the live VMFS datastore to change the name (example, renaming the folder from “SRV01” to “SRV01.bad”. If you renamed the damaged VM, make sure you have enough space for the new restored VM as well.

Procedure:

Mount the VMFS volume on the LUN snapshot to the ESXi host(s)

Identify the VM you want to recover, write it down.
Identify the datastore that the VM resides on, write it down.
Identify the SAN and identify the LUN number that the VMFS datastore resides on, write it down.
Identify the LUN Snapshot unique name/id/number and write it down, confirm the timestamp to make sure it will contain a valid recovery point.
Log on to the SAN and create a host mapping to present the snapshot (you recorded above) to the hosts using a new and unused LUN number.
Log on to your ESXi host and navigate to configuration, then storage adapters.
Select the iSCSI initator and click the “Rescan Storage Adapters” button to rescan all iSCSI LUNs.
VMware ESXi Host Rescan Storage Adapter
Ensure both check boxes are checked and hit “Ok”, wait for the scan to complete (as shown in the “Recent Tasks” window.
VMware ESXi Host Rescan Storage Adapter Window for VMFS Volume and Devices
Now navigate to the “Datastores” tab under configuration, and click on the “Create a new Datastore” button as shown below.
VMware ESXi Host Add Datastore Window
Continue with “VMFS” selected and select continue.
In the next window, you’ll see your existing datastores, as well as your new datastore (from the snapshot). You can leave the “Datastore name” as is since this value will be ignored. In this window you’re going to select the new VMFS datastore from the snapshot. Make sure you confirm this by looking at the LUN number, as well as the value under “SnapshotVolume”. It is critical that you select the snapshot in this window (it should be the new LUN number you added above).
Select next and continue.
On the next window “Mount Option”, you need to change the radio button to and select “Assign a new signature”. This is critical! This will assign a new signature to differentiate it from your existing real production datastore so that the ESXi hosts don’t confuse it.
Continue with the wizard and complete the mount process. At this point ESXi will resignture the VMFS volume and rename it to “snap-OriginalVolumeNameHere”.
You can now browse the VMFS datastore residing on the LUN snapshot and do anything you’d normally be able to do with a normal datastore.

Copy/Move/vMotion the VM from the snapshot VMFS volume to your production VMFS volume

Note: The next steps are only if you are licensed for storage vMotion. If you aren’t you’ll need to use the copy or move function in the file browsing area to copy or move the VMs to your live production VMFS datastores:

Now we’ll go to the vCenter/ESXi host storage area in the web client, and using the “Files” tab, we’ll browse the snapshots VMFS datastore that we just mounted.
Locate the folder for the VM(s) you want to recover, open the folder, right click on the vmx file for the VM and select “Register VM”. Repeat this for any of the VMs you want to recover from the snapshot. Complete the wizard for each VM you register and add it to a host.
Go back to you “Hosts and VMs” view, you’ll now see the VMs are added.
Select and right click on the VM you want to move from the snapshot datastore to your production live datastore, and select “Migrate”.
In the vMotion migrate wizard, select “Change Storage only”.
Continue to the wizard, and storage vMotion the VM from the snapshot VMFS to your production VMFS volume. Wait for the vMotion to complete.
After the storage vMotion is complete, boot the VM and confirm everything is functioning.

Gracefully unmount, detach, and remove the snapshot VMFS from the ESXi host, and then remove the host mapping from the SAN

On each of your ESXi hosts that have access to the SAN, go to the “Datastores” section under the ESXi hosts configuration, right click on the snapshot VMFS datastore, and select “Unmount”. You’ll need to repeat this on each ESXi host that may have automounted the snapshot’s VMFS volume.
On each of your ESXi hosts that have access to the SAN, go to the “Storage Devices” section under the ESXi hosts configuration and identify (by LUN number) the “disk” that is the snapshot LUN. Select and highlight the snapshot LUN disk, select “All Actions” and select “Detach”. Repeat this on each host.
Double check and confirm that the snapshot VMFS datastore (and disk object) have been unmounted and detached from each ESXi host.
You can now log in to your SAN and remove the host mapping for the snapshot-to-LUN. We will not longer present the snapshot LUN to any of the hosts.
Back to the ESXi hosts, navigate to “Storage Adapters”, select the “iSCSI Initiator Adapter”, and click the “Rescan Storage Adapters”. Repeat this for each ESXi host.
VMware ESXi Host Rescan Storage Adapter
You’re done!

Repurpose your old Netbook or Laptop in to the perfect VMware Horizon View VDI Client

Fedora, Linux, VMware, VMware Horizon View No Responses »

Aug 262018

One of the coolest things I love about running VMware Horizon View and VDI is that you can repurpose old computers, laptops, or even netbooks in to perfect VDI clients running Linux! This is extremely easy to do and gives life to old hardware you may have lying around (and we all know there’s nothing wrong with that).

I generally use Fedora and the VMware Horizon View Linux client to accomplish this. See below to see how I do it!

Quick Guide

Download the Fedora Workstation install or netboot ISO from here.
Burn it to a DVD/CD if you have DVD/CD drive, or you can write it to a USB stick using this method here.
Install Fedora on to your laptop/notebook/netbook using the workstation install.
Update your Fedora Linux install using the following command
```
dnf -y upgrade
```

Install the prerequisites for the VMware Horizon View Linux client using these commands

dnf -y install https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
dnf -y install gstreamer-plugins-ugly gstreamer-plugins-bad gstreamer-ffmpeg xine-lib-extras-freeworld xine-lib-extras-freeworld libssl* libcrypto* openssl-devel libpng12 systemd-devel libffi-devel

To fix an issue with package versions and dependancies, run the following commands

ln -s /usr/lib64/libudev.so.1 /usr/lib64/libudev.so.0
ln -s /usr/lib64/libffi.so.6 /usr/lib64/libffi.so.5

Download the VMware Horizon View Linux client from here
Make the VMware bundle executable and then run the installer using these commands (your file name may be different depending on build version number)
```
chmod 777 VMware-Horizon-Client-4.8.0-8518891.x64.bundle
sudo ./VMware-Horizon-Client-4.8.0-8518891.x64.bundle
```
Complete the installation wizard
You’re done!

To run the client, you can find it in the GUI applications list as “VMware Horizon Client”, or you can launch it by running “vmware-view”.

VMware Horizon View on Linux in action

Here is a VMware Horizon View Linux client running on HP Mini 220 Netbook

Additional Notes:

-If you’re comfortable, instead of the workstation install, you can install the Fedora LXQt Desktop spin, which is a lightweight desktop environment perfect for low performance hardware or netbooks. More information and the download link for Fedora LXQt Desktop Spin can be found here: https://spins.fedoraproject.org/en/lxqt/

-If you installed Fedora Workstation and would like to install the LXQt window manager afterwards, you can do so by running the following command (after installing, at login prompt, click on the gear to change window managers):

dnf install @lxqt-desktop-environment

-Some of the prerequisites above in the guide may not be required, however I have installed them anyways for compatibility.

Long term review of the VMware Horizon View Mobile Client for Android

Android, VMware, VMware Horizon View No Responses »

Aug 212018

Well, after using the VMware Horizon Client mobile app (for Android) for a year, I finally decided to do a little write up and review. I use the android client regularly on my Samsung Tab E LTE tablet, and somewhat infrequently on my Samsung Galaxy S9+ mobile phone (due to the smaller screen).

Let’s start off by briefly explaining what VMware Horizon View is, what the client does, and finally the review. I’ll be including a couple screenshots as well to give an idea as to how the interface and resolution looks on the tablet itself.

The VMware Horizon Client mobile app for android is available at: https://play.google.com/store/apps/details?id=com.vmware.view.client.android

What is VMware Horizon View

VMware Hoirzon View is a product and solution that enables VDI technology for a business. VDI stands for Virtual Desktop Infrastructure. When a business uses VDI, they virtualize their desktops and use either thin clients, zero clients, or the view client to access these virtualized desktops. This allows the business to utilize all the awesome technologies that virtualization brings (DRS, High Availability, Backup/DR, high performance, reduced hardware costs) and provide rich computing environments to their users. The technology is also particularly interesting in the fact that it provides amazing remote access capabilities as one can access their desktop very easily with the VMware View Client.

When you tie this on to an advanced security technology such as Duo’s MFA product, you can’t go wrong!

In special case or large environments, enormous cost savings can be realized when implementing VDI.

What is the VMware Horizon View Mobile client for Android

As mentioned above, to access one’s virtualized desktop a client is needed. While a thin client or zero client can be used, this is beyond the scope of this post as here we are only discussing the VMware View client for Android.

You can download the VMware View client for Android from the App store (link here).

The VMware Horizon View Mobile client for Android allows you to connect to your VDI desktop remotely using your Android based phone or tablet. Below is a screenshot I took with my Samsung Tab E LTE tablet (with the side bar expanded):

VMware Horizon View Client on Android Tablet

VMware Horizon View Mobile Client for Android Experience

Please Note: There is more of the review below the screenshots. Scroll down for more!

The app appears to be very lightweight, with an easy interface. Configuration of View Connections Servers, or UAG’s (Unified Access Gateways) is very simple. The login process performs with RADIUS and/or MFA as the desktop client would. In the examples below, you’ll notice I use Duo’s MFA/2FA authentication solution in combination with AD logins.

VMware Horizon View Mobile Client Android Server List

The interface is almost identical to the desktop client with very little differences. The configuration options are also very similar and allow customization of the app, with options for connection quality as an example.

VMware Horizon View Mobile Client Android Server Login

VMware Horizon View Mobile Client Android Login Duo MFA

As you can see above, the RADIUS and Duo Security Login prompts are fully functional.

VMware Horizon View Mobile Client Android Server List

VMware Horizon View Mobile Client Android Windows 10 VDI Desktop

The resolution is perfect for the tablet, and is very usable. The touch interface works extremely well, and text input works as good as it can. While this wouldn’t be used as a replacement for the desktop client, or a thin/zero client, it is a valuable tool for the mobile power user.

With how lightweight and cheap tablets are now, you could almost leave your tablet in your vehicle (although I wouldn’t recommend it), so that in the event of an emergency where you need to access your desktop, you’d be able to using the app.

Pros:

Fluid interface
Windows 10 touch functionality works great
Resolution Support
Samsung Dex is fully supported
Webcam redirection works
Works on Airplanes using in flight WiFi

Cons:

Bandwidth usage
Saving credentials via Fingerprint Scanner would be nice (on the S8+ and S9+)

My Usage

Being in IT, I’ve had to use this many times to log in and manage my vSphere cluster, servers, HPe iLo, check temperatures, and log in to customer environments (I prefer to log in using my VDI desktop, instead of saving client information on the device I’m carrying with me). It’s perfect for these uses.

I also regularly use VDI over LTE. Using VDI over mobile LTE connections works fantastic, however you’ll want to make sure you have an adequate data plan as the H.264 video stream uses a lot of bandwidth. Using this regularly over LTE could cause you to go over your data limits and incur additional charges.

Additional Information

Samsung Dex

The VMware Horizon View Mobile Client for Android also supports Samsung Dex. This means that if you have a Dex dock or the Dex pad, you can use the mobile client to provide a full desktop experience to a monitor/keyboard/mouse using your Samsung Galaxy phone. I’ll be doing a write up later to demo this (it works great).

VMware Horizon Client for Chrome OS

VMware also has a client for Chrome OS, so that you can use your Chromebook to connect to your VDI desktop. You can download VMware Horizon Client for Chrome OS here: https://chrome.google.com/webstore/detail/vmware-horizon-client-for/ppkfnjlimknmjoaemnpidmdlfchhehel

Monitor your VMware vSphere environment remotely with the vSphere Mobile Watchlist Android App

Android, ESXi, VMware, vSphere 2 Responses »

Aug 182018

Did you know that you can monitor and manage your VMware vSphere environment (ESXi hosts, cluster, and VMs) remotely with the “VMware vSphere Mobile Watchlist” app on your Android phone? Well, you can!

Download link: https://play.google.com/store/apps/details?id=com.vmware.beacon

The VMware vSphere Mobile Watchlist (VMware Watchlist) Android App

For some time now, I’ve been using this neat little app from VMware (available for download here) to monitor and manage my vSphere cluster remotely. You can use the app while directly on your LAN, or via VPN (I use it with OpenVPN to connect to my Sophos UTM). I’ve even used it while on airplanes using the on board in-flight WiFi.

The reason why I’m posting about this, is because I’ve never actually heard anyone talk about the app (which I find strange), so I’m assuming others aren’t aware of it’s existence as well.

The app runs extremely well on my Samsung S8+, Samsung S9+, and Samsung Tab E LTE tablet. I haven’t run in to any issues or app crashes.

Let’s take a look at the app

vSphere Mobile Watchlist Login Prompt

The above screen is where you initially log in. I use my Active Directory credentials (since I have my vCenter server integrated with AD).

vSphere Mobile Watchlist Hosts and VM list

vSphere Mobile Watchlist Hosts and VMs

In the default view (shown above), you can view a brief summary of your ESXi hosts, as well as a list of virtual machines running.

vSphere Mobile Watchlist Host Information

After selecting an ESXi host, you can view the hosts resources, details, related objects, as well as flip over to view host options.

vSphere Mobile Watchlist Host Options

Under host options, you can Enter Maintenance mode, reboot the host, shutdown the host, disconnect the host, or view the hosts’ sensor data.

vSphere Mobile Watchlist Host Sensor List and Fan Data

vSphere Mobile Watchlist Host Sensor Data (Fans)

Checking the HPe Proliant DL360p Gen8 fan sensor data with VMware Watchlist.

vSphere Mobile Watchlist Host Sensor Data (Temperature)

Checking the HPe Proliant DL360p Gen8 temperature sensor data with VMware Watchlist. While not shown above, you can select individual items to pull the actual temperature values. Please Note that the temperature values are missing a decimal (Example: 2100 = 21.00 Celsius).

vSphere Mobile Watchlist VM Information View

When selecting a VM (Virtual Machine) from the default view, you can view the VM’s Resources (CPU, Memory, and Storage), Details (IP Addresses, DNS hostnames, Guest OS, VMWare Tools Status), related objects, and a list of other VMs running on the same host.

vSphere Mobile Watchlist VM Options

Flipping over to the VM options, we have the ability to power off, suspend, reset, shutdown, or gracefully restart the VM. We also have some snapshot functionality to take a snapshot, or manage VM snapshots.

Additional Notes

In my environment I have two HPe DL360p Gen8 Servers and the sensor data is fully supported (I used the HPe custom ESXi install image which includes host drivers).

VMware Horizon View: USB Redirection is not available for this desktop

VMware, VMware Horizon View 3 Responses »

Aug 122018

On VMware Horizon view after updating the view agent on the VM, you may notice that USB redirection stops working with the error “USB Redirection is not available for this desktop”. This is due to an issue with the certificates on the VDI host (The VM running the VDI OS), after the VMware view agent upgrade is completed.

To resolve this you must use MMC, open the local computer certificate store, browse to “VMwareView\Certificates”, delete the agent certificates (for the local agent), and finally reboot for the agent to regenerate the certificates.

See below for instructions:

While connected to the VM running the VDI OS, click Start, type “mmc.exe” (without quotations), and open the Microsoft Management Console.
Open MMC by running mmc.exe
Open the “Add/Remote Snap-in” wizard.
Open the Add/Remove Snap-in Wizard
We must now open the local certificate store on the local computer. Select “Certicates” on the Available Snap-ins, click “Add”, select “Computer Account”, then proceed to choose “Local Computer” and complete the wizard.
Select the Computer account certificate store on the local computer
Expand the “Certificates (Local Computer)” on the left underneath “Console Root”. Expand “VMwareView”, then expand and select “Certificates”. Select the certificate on the right that matches the local computer name of the VDI host, right click and select “Delete”. You may have to do this multiple times if multiple certificates exist for the local computer.
Delete the VMwareView local agent certificate
Restart the VDI host. And USB redirection should now be working!
VMware View USB Redirection issue resolved

Cheers to VDI!

Microsoft Windows Server booting to Safe Mode and Directory Services Restore mode with Veeam Backup

ESXi, Microsoft, Veeam Backup and Replication, VMware, Windows Server 6 Responses »

Apr 292018

Running Veeam Backup and Replication, a Microsoft Windows Server Domain Controller may boot in to safe mode and directory services restore mode.

About a week ago, I loaded up Veeam Backup and Replication in to my test environment. It’s a fantastic product, and it’s working great, however today I had a little bit of an issue with a DC running Windows Server 2016 Server Core.

I woke up to a notification that the backup failed due to a VSS snapshot issue. Now I know that VSS can be a little picky at times, so I decided to restart the guest VM. Upon restarting, she came back up, was pingable, and appeared to be running fine, however the backup kept failing with new errors, the event log was looking very strange on the server, and numerous services that were set to automatic were not starting up.

This specific server was installed using Server Core mode, so it has no GUI and is administered via command prompt over RDP, or via remote management utilities. Once RDP’ing in to the server, I noticed the “Safe Mode” branding on each corner of the display, this was very odd. I restarted the server again, this time manually trying to start Active Directory Services manually via services.msc.

This presented:

Event ID: 16652
Source: Directory-Services-SAM
General Description: The domain controller is booting to directory services restore mode.

Screenshot:

The domain controller is booting to directory services restore mode.

This surprised me (and scared me for that matter). I immediately started searching the internet to find out what would have caused this…

To my relief, I read numerous sites that advise that when an active backup is running on a guest VM which is a domain controller, Veeam activates directory services restore mode temporarily, so in the event of a restore, it will boot to this mode automatically. In my case, the switch was not changed back during the backup failure.

Running the following command in a command prompt, verifies that the safeboot switch is set to dsrepair enabled:

bcdedit /v

To disable directory services restore mode, type the following in a command prompt:

bcdedit /deletevalue safeboot

Restart the server and the issue should be resolved!

Older Entries