May 012021
 
Picture of NVMe Storage Server Project

For over a year and a half I have been working on building a custom NVMe Storage Server for my homelab. I wanted to build a high speed storage system similar to a NAS or SAN, backed with NVMe drives that provides iSCSI, NFS, and SMB Windows File Shares to my network.

The computers accessing the NVMe Storage Server would include VMware ESXi hosts, Raspberry Pi SBCs, and of course Windows Computers and Workstations.

The focus of this project is on high throughput (in the GB/sec) and IOPS.

The current plan for the storage environment is for video editing, as well as VDI VM storage. This can and will change as the project progresses.

The History

More and more businesses are using all-flash NVMe and SSD based storage systems, so I figured there’s no reason why I can’t have build and have my own budget custom all NVMe flash NAS.

This is the story of how I built my own NVMe based Storage Server.

The first version of the NVMe Storage Server consisted of the IO-PEX40152 card with 4 x 2TB Sabrent Rocket 4 NVMe drives inside of an HPE Proliant DL360p Gen8 Server. The server was running ESXi with TrueNAS virtualized, and the PCIe card passed through to the TrueNAS VM.

The results were great, the performance was amazing, and both servers had access to the NFS export via 2 x 10Gb SFP+ networking.

There were three main problems with this setup:

  1. Virtualized – Once a month I had an ESXi PSOD. This was either due to overheating of the IO-PEX40152 card because of modifications I made, or bugs with the DL360p servers and PCIe passthrough.
  2. NFS instead of iSCSI – Because TrueNAS was virtualized inside of the host that was using it for storage, I had to use NFS since the host virtualizing TrueNAS would also be accessing the data on the TrueNAS VM. When shutting down the host, you need to shut down TrueNAS first. NFS disconnects are handled way healthier than iSCSI disconnects (which can cause corruption even if no files are being used).
  3. CPU Cores maxed on data transfer – When doing initial testing, I was maxing out the CPU cores assigned to the TrueNAS VM because the data transfers were so high. I needed a CPU and setup that was better fit.

Version 1 went great, but you can see some things needed to be changed. I decided to go with a dedicated server, not virtualize TrueNAS, and go for a newer CPU with a higher Ghz speed.

And so, version 2 was born (built). Keep reading and scrolling for pictures!

The Hardware

On version 2 of the project, the hardware includes:

Notes on the Hardware:

  • While the ML310e Gen8 v2 server is a cheap low entry server, it’s been a fantastic team member of my homelab.
  • HPE Dual 10G Port 560SFP+ adapters can be found brand new in unsealed boxes on eBay at very attractive prices. Using HPE Parts inside of HPE Servers, avoids the fans from spinning up fast.
  • The ML310e Gen8 v2 has some issues with passing through PCIe cards to ESXi. Works perfect when not passing through.

The new NVMe Storage Server

I decided to repurpose an HPE Proliant ML310e Gen8 v2 Server. This server was originally acting as my Nvidia Grid K1 VDI server, because it supported large PCIe cards. With the addition of my new AMD S7150 x2 hacked in/on to one of my DL360p Gen8’s, I no longer needed the GRID card in this server and decided to repurpose it.

Picture of an HPe ML310e Gen8 v2 with NVMe Storage
HPe ML310e Gen8 v2 with NVMe Storage

I installed the IOCREST IO-PEX40152 card in to the PCIe 16x slot, with 4 x 2TB Sabrent Rocket 4 NVME drives.

Picture of IOCREST IO-PEX40152 with GLOTRENDS M.2 NVMe SSD Heatsink on Sabrent Rocket 4 NVME
IOCREST IO-PEX40152 with GLOTRENDS M.2 NVMe SSD Heatsink on Sabrent Rocket 4 NVME

While the server has a PCIe 16x wide slot, it only has an 8x bus going to the slot. This means we will have half the capable speed vs the true 16x slot. This however does not pose a problem because we’ll be maxing out the 10Gb NICs long before we max out the 8x bus speed.

I also installed an HPE Dual Port 560SFP+ NIC in to the second slot. This will allow a total of 2 x 10Gb network connections from the server to the Ubiquiti UniFi US-16-XG 10Gb network switch, the backbone of my network.

The Server also have 4 x Hot Swappable HD bays on the front. When configured in HBA mode (via the BIOS), these are accessible by TrueNAS and can be used. I plan on populating these with 4 x 4TB HPE MDL SATA Hot Swappable drives to act as a replication destination for the NVMe pool and/or slower magnetic long-term storage.

Front view of HPE ML310e Gen8 v2 with Hotswap Drive bays
HPE ML310e Gen8 v2 with Hotswap Drive bays

I may also try to give WD RED Pro drives a try, but I’m not sure if they will cause the fans to speed up on the server.

TrueNAS Installation and Configuration

For the initial Proof-Of-Concept for version 2, I decided to be quick and dirty and install it to a USB stick. I also waited until I installed TrueNAS on to the USB stick and completed basic configuration before installing the Quad NVMe PCIe card and 10Gb NIC. I’m using a USB 3.0 port on the back of the server for speed, as I can’t verify if the port on the motherboard is USB 2 or USB 3.

Picture of a TrueNAS USB Stick on HPE ML310e Gen8 v2
TrueNAS USB Stick on HPE ML310e Gen8 v2

TrueNAS installation worked without any problems whatsoever on the ML310e. I configured the basic IP, time, accounts, and other generic settings. I then proceeded to install the PCIe cards (storage and networking).

Screenshot of TrueNAS Dashboard Installed on NVMe Storage Server
TrueNAS Installed on NVMe Storage Server

All NVMe drives were recognized, along with the 2 HDDs I had in the front Hot-swap bays (sitting on an HP B120i Controller configured in HBA mode).

Screenshot of available TrueNAS NVMe Disks
TrueNAS NVMe Disks

The 560SFP+ NIC also was detected without any issues and available to configure.

Dashboard Screenshot of TrueNAS 560SFP+ 10Gb NIC
TrueNAS 560SFP+ 10Gb NIC

Storage Configuration

I’ve already done some testing and created a guide on FreeNAS and TrueNAS ZFS Optimizations and Considerations for SSD and NVMe, so I made sure to use what I learned in this version of the project.

I created a striped pool (no redundancy) of all 4 x 2TB NVMe drives. This gave us around 8TB of usable high speed NVMe storage. I also created some datasets and a zVOL for iSCSI.

Screenshot of NVMe TrueNAS Storage Pool with Datasets and zVol
NVMe TrueNAS Storage Pool with Datasets and zVol

I chose to go with the defaults for compression to start with. I will be testing throughput and achievable speeds in the future. You should always test this in every and all custom environments as the results will always vary.

Network Configuration

Initial configuration was done via the 1Gb NIC connection to my main LAN network. I had to change this as the 10Gb NIC will be directly connected to the network backbone and needs to access the LAN and Storage VLANs.

I went ahead and configured a VLAN Interface on VLAN 220 for the Storage network. Connections for iSCSI and NFS will be made on this network as all my ESXi servers have vmknics configured on this VLAN for storage. I also made sure to configure an MTU of 9000 for jumbo frames (packets) to increase performance. Remember that all hosts must have the same MTU to communicate.

Screenshot of 10Gb NIC on Storage VLAN
10Gb NIC on Storage VLAN

Next up, I had to create another VLAN interface for the LAN network. This would be used for management, as well as to provide Windows File Share (SMB/Samba) access to the workstations on the network. We leave the MTU on this adapter as 1500 since that’s what my LAN network is using.

Screenshot of 10Gb NIC on LAN VLAN
10Gb NIC on LAN VLAN

As a note, I had to delete the configuration for the existing management settings (don’t worry, it doesn’t take effect until you hit test) and configure the VLAN interface for my LANs VLAN and IP. I tested the settings, confirmed it was good, and it was all setup.

At this point, only the 10Gb NIC is now being used so I went ahead and disconnected the 1Gb network cable.

Sharing Setup and Configuration

It’s now time to configure the sharing protocols that will be used. As mentioned before, I plan on deploying iSCSI, NFS, and Windows File Shares (SMB/Samba).

iSCSI and NFS Configuration

Normally, for a VMware ESXi virtualization environment, I would always usually prefer iSCSI based storage, however I also wanted to configure NFS to test throughput of both with NVMe flash storage.

Earlier, I created the datasets for all my my NFS exports and a zVOL volume for iSCSI.

Note, that in order to take advantage of the VMware VAAI storage directives (enhancements), you must use a zVOL to present an iSCSI target to an ESXi host.

For NFS, you can simply create a dataset and then export it.

For iSCSI, you need to create a zVol and then configure the iSCSI Target settings and make it available.

SMB (Windows File Shares)

I needed to create a Windows File Share for file based storage from Windows computers. I plan on using the Windows File Share for high-speed storage of files for video editing.

Using the dataset I created earlier, I configured a Windows Share, user accounts, and tested accessing it. Works perfect!

Connecting the host

Connecting the ESXi hosts to the iSCSI targets and the NFS exports is done in the exact same way that you would with any other storage system, so I won’t be including details on that in this post.

We can clearly see the iSCSI target and NFS exports on the ESXi host.

Screenshot of TrueNAS NVMe iSCSI Target on VMware ESXi Host
TrueNAS NVMe iSCSI Target on VMware ESXi Host
Screenshot of NVMe iSCSI and NFS ESXi Datastores
NVMe iSCSI and NFS ESXi Datastores

To access Windows File Shares, we log on and map the network share like you would normally with any file server.

Testing

For testing, I moved (using Storage vMotion) my main VDI desktop to the new NVMe based iSCSI Target LUN on the NVMe Storage Server. After testing iSCSI, I then used Storage vMotion again to move it to the NFS datastore. Please see below for the NVMe storage server speed test results.

Speed Tests

Just to start off, I want to post a screenshot of a few previous benchmarks I compiled when testing and reviewing the Sabrent Rocket 4 NVMe SSD disks installed in my HPE DL360p Gen8 Server and passed through to a VM (Add NVMe capability to an HPE Proliant DL360p Gen8 Server).

Screenshot of CrystalDiskMark testing an IOCREST IO-PEX40152 and Sabrent Rocket 4 NVME SSD for speed
CrystalDiskMark testing an IOCREST IO-PEX40152 and Sabrent Rocket 4 NVME SSD
Screenshot of CrystalDiskMark testing IOPS on an IOCREST IO-PEX40152 and Sabrent Rocket 4 NVME SSD
CrystalDiskMark testing IOPS on an IOCREST IO-PEX40152 and Sabrent Rocket 4 NVME SSD

Note, that when I performed these tests, my CPU was maxed out and limiting the actual throughput. Even then, these are some fairly impressive speeds. Also, these tests were directly testing each NVMe drive individually.

Moving on to the NVMe Storage Server, I decided to test iSCSI NVMe throughput and NFS NVMe throughput.

I opened up CrystalDiskMark and started a generic test, running a 16GB test file a total of 6 times on my VDI VM sitting on the iSCSI NVMe LUN.

Screenshot of NVMe Storage Server iSCSI Benchmark with CrystalDiskMark
NVMe Storage Server iSCSI Benchmark with CrystalDiskMark

You can see some impressive speeds maxing out the 10Gb NIC with crazy performance of the NVME storage:

  • 1196MB/sec READ
  • 1145.28MB/sec WRITE (Maxing out the 10GB NIC)
  • 62,725.10 IOPS READ
  • 42,203.13 IOPS WRITE

Additionally, here’s a screenshot of the ix0 NIC on the TrueNAS system during the speed test benchmark: 1.12 GiB/s.

Screenshot of TrueNAS NVME Maxing out 10Gig NIC
TrueNAS NVME Maxing out 10Gig NIC

And remember this is with compression. I’m really excited to see how I can further tweak and optimize this, and also what increases will come with configuring iSCSI MPIO. I’m also going to try to increase the IOPS to get them closer to what each individual NVMe drive can do.

Now on to NFS, the results were horrible when moving the VM to the NFS Export.

Screenshot of NVMe Storage Server NFS Benchmark with CrystalDiskMark
NVMe Storage Server NFS Benchmark with CrystalDiskMark

You can see that the read speed was impressive, but the write speed was not. This is partly due to how writes are handled with NFS exports.

Clearly iSCSI is the best performing method for ESXi host connectivity to a TrueNAS based NVMe Storage Server. This works perfect because we’ll get the VAAI features (like being able to reclaim space).

iSCSI MPIO Speed Test

This is more of an update… I was finally able to connect, configure, and utilize the 2nd 10Gbe port on the 560SFP+ NIC. In my setup, both hosts and the TrueNAS storage server all have 2 connections to the switch, with 2 VLANs and 2 subnets dedicated to storage. Check out the before/after speed tests with enabling iSCSI MPIO.

As you can see I was able to essentially double my read speeds (again maxing out the networking layer), however you’ll notice that the write speeds maxed out at 1598MB/sec. I believe we’ve reached a limitation of the CPU, PCIe bus, or something else inside of the server. Note, that this is not a limitation of the Sabrent Rocket 4 NVME drives, or the IOCREST NVME PCIe card.

Moving Forward

I’ve had this configuration running for around a week now with absolutely no issues, no crashes, and it’s been very stable.

Using a VDI VM on NVMe backed storage is lightning fast and I love the experience.

I plan on running like this for a little while to continue to test the stability of the environment before making more changes and expanding the configuration and usage.

Future Plans (and Configuration)

  • Drive Bays
    • I plan to populate the 4 hot-swappable drive bays with HPE 4TB MDL drives. Configured with RaidZ1, this should give me around 12TB usable storage. I can use this for file storage, backups, replication, and more.
  • NVMe Replication
    • This design was focused on creating non-redundant extremely fast storage. Because I’m limited to a total of 4 NVMe disks in this design, I chose not to use RaidZ and striped the data. If one NVMe drive is lost, all data is lost.
    • I don’t plan on storing anything important, and at this point the storage is only being used for VDI VMs (which are backed up), and Video editing.
    • If I can populate the front drive bays, I can replicate the NVMe storage to the traditional HDD storage on a frequent basis to protect against failure to some level or degree.
  • Version 3 of the NVMe Storage Server
    • More NVMe and Bigger NVMe – I want more storage! I want to test different levels of RaidZ, and connect to the backbone at even faster speeds.
    • NVME Drives with PLP (Power Loss Prevention) for data security and protection.
    • Dual Power Supply

Let me know your thoughts and ideas on this setup!

Apr 252021
 
Screenshot of a Hybrid Azure AD Joined login

If you’re using Azure AD, and have Hybrid Azure AD joined machines, special considerations must be made with non-persistent VDI workstations and VMs. This applies to Instant Clones on VMware Horizon.

Due to the nature of non-persistent VDI, machines are created and destroyed on the fly with a user getting an entirely new workstation on every login.

Hybrid Azure AD joined workstations not only register on the local domain Active Directory, but also register on the Azure AD (Azure Active Directory).

The Problem

If you have Hybrid Azure AD configured and machines performing the Hybrid Join, this will cause numerous machines to be created on Azure AD, in a misconfigured and/or unregistered state. When the non-persistent instant clone is destroyed and re-created, it will potentially have the same computer name as a previous machine, but will be unable to utilize the existing registration.

This conflict state could potentially make your Azure AD computer OU a mess.

VMware Horizon 8 version 2303 now supports Hybrid Azure AD joined non-persistent instant clones using Azure AD Connect. If you are using an older version, or using a different platform for non-persistent VDI, you’ll need to reference the solution below.

The Solution

Please see below for a few workarounds and/or solutions:

  1. Upgrade to VMware Horizon 8 2303
  2. Use Seamless SSO instead of Hybrid Azure AD join (click here for more information)
  3. Utilize login/logoff scripts to Azure AD join and unjoin on user login/logoff. You may have to create a cleanup script to remove old/stale records from Azure AD as this can and will create numerous computer accounts on Azure AD.
  4. Do not allow non-persistent virtual machines to Hybrid Domain Join. This can be accomplished either by removing the non-persistent VDI computer OU from synchronization with Azure AD Connect (OU Filtering information at https://docs.microsoft.com/en-us/azure/active-directory/hybrid/how-to-connect-sync-configure-filtering) or by disabling the scheduled task to perform an Azure AD join.

In my environment I elected to remove the non-persistent computer OU from Azure AD Connect sync, and it’s been working great. It also keeps my Azure Active Directory nice and clean.

Jan 172021
 

After upgrading from Horizon 8 2006 to Horizon 8 2012, audio stopped working. When connected to a VDI session, audio is not being passed through to the client.

The Problem

Audio simply does not work. Using the Chrome and multimedia redirection, audio will work, but this is most likely due to the fact the client is handling multimedia.

The Fix

Removing the audio drivers (forcing uninstall/deleting the audio driver) and re-installing the agent does not correct this.

Uninstalling and reinstalling the Horizon Client does not correct this.

Audio does function on the Horizon Android client so I isolated this to the Windows client.

After further troubleshooting, I opened the Windows Sound mixer (Right click on the audio icon in the system tray, select “Open Volume Mixer”). I noticed that not only was the VMware Horizon client at 0, but it was also muted.

VMware Horizon View Client Audio Mixer

Unmuting this and raising the volume slider resolved the issue.

Dec 072020
 
Picture of a business office with cubicles

In this post I’m going to explain what VDI is in the most simplest form and how you can benefit from using virtual desktop infrastructure (virtualized desktops) in your EUC strategy.

Virtual Desktop Infrastructure (VDI)

VDI standards for Virtual Desktop Infrastructure. Think of your existing physical desktop infrastructure (your desktop computers, also called end user computing), now virtualize those desktop computers in a virtual environment much like your servers are, and you now have Virtual Desktop Infrastructure.

End User Computing (EUC)

Traditionally end user computing has been delivered by means of deploying physical (real) computers to each user in your office (and possibly remote users). This brings with it the cost of the systems, the time/cost to maintain the systems and hardware, and the management overhead of maintaining those systems.

By utilizing VDI, you can significantly reduce the cost, management, and maintenance required to maintain your EUC infrastructure.

What is VDI

When you implement a VDI solution, you virtualize your desktops and workstations on a virtualization server, much like your servers are probably already virtualized. Users will connect via software, a thin client, or a zero client to establish the session to transmit and receive the video, monitor, and keyboard of workstation that is virtualized.

This might sound familiar, like RDS (Remote Desktop Services). However, in an RDS environment numerous users share the same server and resources and access it un a multi-user fashion, whereas with VDI they are using a virtualized Windows instance dedicated to them running an OS like Windows 10.

How does VDI work

Using the software, thin client, or zero client, a user establishes a session to a connection broker, which then passes it along to the Virtual Machine running on the server. The Virtual Machine encodes and compresses the graphics and then connects the users keyboard and mouse to the VM.

What’s even cooler, is that remote devices like printers and USB devices can also be forwarded on to the VM, giving the user the feeling that the computer that’s running on the server, is actually right in front of them.

And if that isn’t cool enough, in an environment where 3D accelerated and high-performance graphics are required, you can use special graphics cards and GPUs to provide those high end graphics remotely to users. Technically you could game, do engineering work, video and graphics editing, and more.

Why use VDI

So your desktops are now virtualized. This means you no longer need to maintain numerous physical PCs and the hardware that is inside of them.

You can deploy a standardized golden image that instantly clones as users log in to give them a pre-configured and maintained environment. This means you manage 1 or few desktops which can get deployed to hundreds of users, instead of managed hundreds of desktops.

If a thin client or zero client fails you can simply re-deploy a new unit to the user, which are very inexpensive, and reduces downtime.

In the event of a disaster, your VDI EUC environment would be integrated in to your disaster recovery solution, meaning it would be very easy to get users back up and running.

One of the best parts is that the environment can be used inside of your office and externally, allowing you to provide a smooth experience for remote users. This made business continuity a breeze for organizations that need to deploy remote users or “Work from home” users on the fly.

The cost of VDI

The cost to roll out a VDI solution varies depending on the number of users, types of users, and functionality you’d like.

Typically, VDI is a no-brainer for large organizations and enterprises due to the cost savings on hardware, management, and maintaining the solution vs traditional desktops. But smaller organizations can also benefit from VDI, examples being organizations that use expensive desktops and/or laptops for uses such as engineering, software development, and other uses that require high-cost workstations.

One last thought I want to leave you with; imagine an environment with 50-100 systems, and all the wasted power and CPU cycles when users are just browsing the internet. In a virtual environment you can over-allocate resources, which means you can identify user trends and only purchase the hardware you need to based on observed workloads. This can significantly reduce the cost of hardware, especially for software development, engineering, and other high performance computing.

For more information on VDI, take a look at my other VDI related blog posts.

Oct 232020
 
vCSA Update Installation

When updating VMware vCenter vCSA 7.0 U1 (Build 16858589) to vCSA 7.0 U1 (Build 17004997/17005016, Version 7.0.1.00100), you may notice that the update fails and reports issues with pre-update checks.

Pre-update checks done prior to the update will pass and allow you to proceed, however it’s the installation that will fail and crash reporting this error.

After the installation fails, you will no longer be able to log in to the vCSA VAMI reporting the error “Unable to Login” using the root account.

You are able to login via SSH. Resetting the root password via SSH will not resolve this issue.

The Problem

In the past, issues with the root password expiring have caused similar behavior on the vCSA VAMI. Changing the root password does not resolve this specific issue.

Further troubleshooting, it appears that special characters in the root password such as “!”, “.”, and “@” caused this issue to occur in my environment.

I was not able to fix the broken vCSA after the failed update. Access to the vCSA was not possible, however vCenter functions were still operating.

The Solution

To resolve this situation in my environment, I restored a snapshot of the vCSA taken prior to updating.

After restoring the snapshot, I changed the root password for VAMI and restarted the vCSA.

Another snapshot was taken prior to attempting the upgrade, which was now succesfull after removing special characters out of the root password.

Oct 152020
 
VMware vCLS VM in VM List

Did a new VM appear on your VMware vSphere cluster called “vCLS”? Maybe multiple appeared named “vCLS (1)”, “vCLS (2)”, “vCLS (3)” appeared.

VMware vCLS VM in vSphere Cluster Objects
VMware vCLS VM in vSphere Cluster Objects

This could be frightening but fear not, this is part of VMware vSphere 7.0 Update 1.

What is the vCLS VM?

The vCLS virtural machine is essentially an “appliance” or “service” VM that allows a vSphere cluster to remain functioning in the event that the vCenter Server becomes unavailable. It will maintain the health and services of that cluster.

Where did the vCLS VM come from?

The vCLS VM will appear after upgrading to vSphere 7.0 Update 1. I’m assuming it was deployed during the upgrade process.

It does not appear in the standard Cluster, Hosts, and VMs view, but does appear when looking at the vSphere objects VM lists, Storage VM lists, etc…

Is it normal to have more than one vCLS VM?

The vCLS VMs are created when hosts are added to a vSphere Cluster. Up to 3 vCLS VMs are required to run in each vSphere Cluster.

The vCLS VMs will also appear on clusters which contain only one or two hosts. These configurations will result in either 1 or 2 vCLS VMs named “vCLS (1)” and “vCLS (2)”.

A note on licensing in regards to the vCLS VM

For VMware environments that use VM based licensing like vSphere for ROBO (Remote Office Branch Office), vCLS VMs are shown in the licensing interface as counting towards licensed VMs. Please Note that these VMs do not official count towards your purchased licenses as these are VMware System VMs. Please read VMware KB 80472 for more information on this.

More Information on vCLS VMs

For more information and technical specifics, you can visit the link below:

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenterhost.doc/GUID-96BD6016-4BE7-4B1C-8269-568D1555B08C.html

Hope this posts helps, and puts some minds at ease. Your VMware environment has NOT been compromised.

Oct 102020
 

If you’re like me and use an older Nvidia GRID K1 or K2 vGPU video card for your VDI homelab, you may notice that when using VMware Horizon that VMware Blast h264 encoding is no longer being offloaded to the GPU and is instead being encoded via the CPU.

The Problem

Originally when an environment was configured with an Nvidia GRID K1 or K2 card, not only does the card provide 3D acceleration and rendering, but it also offloads the VMware BLAST h264 stream (the visual session) so that the CPU doesn’t have to. This results in less CPU usage and provides a streamlined experience for the user.

This functionality was handled via NVFBC (Nvidia Frame Buffer Capture) which was part of the Nvidia Capture SDK (formerly known as GRID SDK). This function allowed the video card to capture the video frame buffer and encode it using NVENC (Nvidia Encoder).

Ultimately after spending hours troubleshooting, I learned that NVFBC has been deprecated and is no longer support, hence why it’s no longer functioning. I also checked and noticed that tools (such as nvfbcenable) were no longer bundled with the VMware Horizon agent. One can assume that the agent doesn’t even attempt to check or use this function.

Symptoms

Before I was aware of this, I noticed that while 3D Acceleration and graphics were functioning, I was experiencing high CPU usage. Upon further investigation I noticed that my VMware BLAST sessions were not offloading h264 encoding to the video card.

VMware Horizon Performance Tracker
VMware Horizon Performance Tracker with NVidia GRID K1

You’ll notice above that under the “Encoder” section, the “Encoder Name” was listed as “h264 4:2:0”. Normally this would say “NVIDIA NvEnc H264” (or “NVIDIA NvEnc HEVC” on newer cards) if it was being offloaded to the GPU.

Looking at a VMware Blast session (Blast-Worker-SessionId1.log), the following lines can be seen.

[INFO ] 0x1f34 bora::Log: NvEnc: VNCEncodeRegionNvEncLoadLibrary: Loaded NVIDIA SDK shared library "nvEncodeAPI64.dll"
[INFO ] 0x1f34 bora::Log: NvEnc: VNCEncodeRegionNvEncLoadLibrary: Loaded NVIDIA SDK shared library "nvml.dll"
[WARN ] 0x1f34 bora::Warning: GetProcAddress: Failed to resolve nvmlDeviceGetEncoderCapacity: 127
[WARN ] 0x1f34 bora::Warning: GetProcAddress: Failed to resolve nvmlDeviceGetProcessUtilization: 127
[WARN ] 0x1f34 bora::Warning: GetProcAddress: Failed to resolve nvmlDeviceGetGridLicensableFeatures: 127
[INFO ] 0x1f34 bora::Log: NvEnc: VNCEncodeRegionNvEncLoadLibrary: Some NVIDIA nvml functions unavailable, unloading
[INFO ] 0x1f34 bora::Log: NvEnc: VNCEncodeRegionNvEncUnloadLibrary: Unloading NVIDIA SDK shared library "nvEncodeAPI64.dll"
[INFO ] 0x1f34 bora::Log: NvEnc: VNCEncodeRegionNvEncUnloadLibrary: Unloading NVIDIA SDK shared library "nvml.dll"
[WARN ] 0x1f34 bora::Warning: GetProcAddress: Failed to resolve nvmlDeviceGetEncoderCapacity: 127
[WARN ] 0x1f34 bora::Warning: GetProcAddress: Failed to resolve nvmlDeviceGetProcessUtilization: 127
[WARN ] 0x1f34 bora::Warning: GetProcAddress: Failed to resolve nvmlDeviceGetGridLicensableFeatures: 127

You’ll notice it tries to load the proper functions, however it fails.

The Solution

Unfortunately the only solution is to upgrade to newer or different hardware.

The GRID K1 and GRID K2 cards have reached their EOL (End of Life) and are no longer support. The drivers are not being maintained or updated so I doubt they will take advantage of the newer frame buffer capture functions of Windows 10.

Newer hardware and solutions have incorporated this change and use a different means of frame buffer capture.

To resolve this in my own homelab, I plan to migrate to an AMD FirePro S7150x2.

Jul 222020
 
VMware Logon

When upgrading from any version of VMware vCSA to version 7.0, you may encounter a problem during the migration phase and be asked to specifiy a new “Export Directory”.

I’ve seen this occur on numerous upgrades and often find the same culprit causing the issue. I’ve found a very simple fix compared to other solutions online.

The full prompt for this issue is: “Enter a new export directory on the source machine below”

The Problem

When you upgrade the vCenter vCSA, the process migrates all data over from the source appliance, to the new vCSA 7 appliance.

This data can include the following (depending on your selection):

  • Configuration
  • Configuration and historical data (events and tasks)
  • Configuration and historical data (events, tasks, and performance metric)
  • vSphere Update Manager (updates, configuration, etc.)

This data can accumulate, especially the VMware vSphere Update Manager.

In the most recent upgrade I performed, I noticed that the smallest option (configuration only) was around 8GB, which is way over the 4.7GB default limit.

Could it be vSphere Update Manager?

I’ve seen VMware VUM cause numerous issues over the years with upgrades. VUM has caused issues upgrading from earlier versions to 6.x, and in this case it caused this issue upgrading to vCSA 7.x as well.

In my diagnosis, I logged in to the SSH console of the source appliance, and noticed that the partitions containing the VUM data (which includes update files) was around 7.4GB. This is the “/storage/updatemgr/” partition.

I wasn’t sure if this was included, but the 8GB of configuration, minus the 7.4GB of VUM data, could technically get me to around 0.6GB for migration if this was in fact included.

In my environment, I have the default (and simple) implementation of VUM with the only customization being the HPE VIBs depot. I figured maybe I should blast away the VUM and start from scratch on VMware vCSA 7.0 to see if this fixes the issue.

The Fix

To fix this issue, I simply completely reset the VMware Update Manager Database.

For details on this process and before performing these steps, please see VMware KB 2147284.

Let’s get to it:

  1. Close the migration window (you can reopen this later)
  2. Log in to your vCSA source appliance via SSH or console
  3. Run the applicable steps as defined in the VMware KB 2147284 to reset VUM (WARNING: commands are version specific). In my case on vCSA 6.5 I ran the following commands:
    1. shell
    2. service-control --stop vmware-updatemgr
    3. /usr/lib/vmware-updatemgr/bin/updatemgr-util reset-db
    4. rm -rf /storage/updatemgr/patch-store/*
    5. service-control --start vmware-updatemgr
  4. Open your web broswer and navigate to https://new-vcsa-IP:5480 and resume the migration. You will now notice a significant space reduction and won’t need to specify a new mount point.

That’s it! You have a shiny new clean VUM instance, and can successfully upgrade to vCSA 7.0 without having to specify a new mount point.

To reconfigure and restore any old configuration to VUM, you’ll do so in the “VMware Lifecycle Management” section of the VMware vCenter Server Appliance interface.

Alternatively, in the rare event it’s not related to the VUM data, you can set the export directory to somewhere in “/tmp/” which is another workaround this issue which may allow you to continue.

Jul 132020
 
Picture of the DUO Security Logo

When you’re looking for additional or enhanced options to secure you’re business and enterprise IT systems, MFA/2FA can help you achieve this. Get away from the traditional single password, and implement additional means of authentication! MFA provides a great compliment to your cyber-security policies.

My company, Digitally Accurate Inc, has been using the Duo Security‘s MFA product in our own infrastructure, as well as our customers environments for some time. Digitally Accurate is a DUO Partner and can provide DUO MFA Services including licensing/software and the hardware tokens (Duo D-100 Tokens using HOTP).

What is MFA/2FA

MFA is short for Multi Factor authentication, additionally 2FA is short for Two Factor Authentication. While they are somewhat the same, multi means many, and 2 means two. Additional security is provided with both, since it provides more means of authentication.

Traditionally, users authenticate with 1 (one) level of authentication: their password. In simple terms MFA/2FA in addition to a password, provides a 2nd method of authentication and identity validation. By requiring users to authentication with a 2nd mechanism, this provides enhanced security.

Why use MFA/2FA

In a large portion of security breaches, we see users passwords become compromised. This can happen during a phishing attack, virus, keylogger, or other ways. Once a malicious user or bot has a users credentials (username and password), they can access resources available to that user.

By implementing a 2nd level of authentication, even if a users password becomes compromised, the real (or malicious user) must pass a 2nd authentication check. While this is easy for the real user, in most cases it’s nearly impossible for a malicious user. If a password get’s compromised, nothing can be accessed as it requires a 2nd level of authentication. If this 2nd method is a cell phone or hardware token, a malicious user won’t be ale to access the users resources unless they steal the cell phone, or hardware token.

How does MFA/2FA work

When deploying MFA or 2FA you have the option of using an app, hardware token (fob), or phone verification to perform the additional authentication check.

After a user attempts to logs on to a computer or service with their username and password, the 2nd level of authentication will be presented, and must pass in order for the login request to succeed.

Please see below for an example of 2FA selection screen after a successful username and password:

Screenshot of Duo MFA 2FA Prompt on Windows Login
Duo Security Windows Login MFA 2FA Prompt

After selecting an authentication method for MFA or 2FA, you can use the following

2FA with App (Duo Push)

Duo Push sends an authentication challenge to your mobile device which a user can then approve or deny.

Please see below for an example of Duo Push:

Screenshot of Duo Push Notification to Mobile Android App
Duo Push to Mobile App on Android

Once the user selects to approve or deny the login request, the original login will either be approved or denied. We often see this as being the preferred MFA/2FA method.

2FA with phone verification (Call Me)

Duo phone verification (Call Me) will call you on your phone number (pre-configured by your IT staff) and challenge you to either hangup to deny the login request, or press a button on the keypad to accept the login request.

While we rarely use this option, it is handy to have as a backup method.

2FA with Hardware Token (Passcode)

Duo Passcode challenges are handled using a hardware token (or you can generate a passcode using the Duo App). Once you select this method, you will be prompted to enter the passcode to complete the 2FA authentication challenge. If you enter the correct passcode, the login will be accepted.

Here is a Duo D-100 Token that uses HOTP (HMAC-based One Time Password):

Picture of Duo D-100 HOTP Hardware Token
Duo D-100 HOTP Hardware Token

When you press the green button, a passcode will be temporarily displayed on the LCD display which you can use to complete the passcode challenge.

You can purchase Hardware Token’s directly from Digitally Accurate Inc by contacting us, your existing Duo Partner, or from Duo directly. Duo is also compatible with other 3rd party hardware tokens that use HOTP and TOTP.

2FA with U2F

While you can’t visibly see the option for U2F, you can use U2F as an MFA or 2FA authentication challenge. This includes devices like a Yubikey from Yubico, which plugs in to the USB port of your computer. You can attach a Yubikey to your key chain, and bring it around with you. The Yubikey simply plugs in to your USB port and has a button that you press when you want to authenticate.

When the 2FA window pops up, simply hit the button and your Yubikey will complete the MFA/2FA challange.

What can MFA/2FA protect

Duo MFA supports numerous cloud and on-premise applications, services, protocols, and technologies. While the list is very large (full list available at https://duo.com/product/every-application), we regularly deploy and use Duo Security for the following configurations.

Windows Logins (Server and Workstation Logon)

Duo MFA can be deployed to not only protect your Windows Servers and Workstations, but also your remote access system as well.

When logging on to a Windows Server or Windows Workstation, a user will be presented with the following screen for 2FA authentication:

Screenshot of Duo MFA 2FA Prompt on Windows Login
Duo Security Windows Login MFA 2FA Prompt

Below you can see a video demonstration of DUO on Windows Login.

DUO works with both Windows Logins and RDP (Remote Desktop Protocol) Logins.

VMWare Horizon View Clients (VMWare VDI Logon)

Duo MFA can be deployed to protect your VDI (Virtual Desktop Infrastructure) by requiring MFA or 2FA when users log in to access their desktops.

When logging on to the VMware Horizon Client, a user will be presented with the following screen for 2FA authentication:

Screenshot of Duo MFA 2FA Prompt on VMWare Horizon Client Login
Duo Security VMWare Horizon Client Login MFA 2FA Prompt

Below you can see a video demonstration of DUO on VMware Horizon View (VDI) Login.

Sophos UTM (Admin and User Portal Logon)

Duo MFA can be deployed to protect your Sophos UTM firewall. You can protect the admin account, as well as user accounts when accessing the user portal.

If you’re using the VPN functionality on the Sophos UTM, you can also protect VPN logins with Duo MFA.

Unix and Linux (Server and Workstation Logon)

Duo MFA can be deployed to protect your Unix and Linux Servers. You can protect all user accounts, including the root user.

We regularly deploy this with Fedora and CentOS (even FreePBX) and you can protect both SSH and/or console logins.

When logging on to a Unix or Linux server, a user will be presented with the following screen for 2FA authentication:

Screenshot of Duo MFA 2FA Prompt on CentOS Linux Login
Duo Security CentOS Linux login MFA 2FA Prompt

Below you can see a video demonstration of DUO on Linux.

WordPress Logon

Duo MFA can be deployed to protect your WordPress blog. You can protect your admin and other user accounts.

If you have a popular blog, you know how often bots are attempting to hack and brute force your passwords. If by chance your admin password becomes compromised, using MFA or 2FA can protect your site.

When logging on to a WordPress blog admin interface, a user will be presented with the following screen for 2FA authentication:

Screenshot of Duo MFA 2FA Prompt on WordPress Login
Duo Security WordPress Login MFA 2FA Prompt

Below you can see a video demonstration of DUO on a WordPress blog.

How easy is it to implement

Implementing Duo MFA is very easy and works with your existing IT Infrastructure. It can easily be setup, configured, and maintained on your existing servers, workstations, and network devices.

Duo offers numerous plugins (for windows), as well as options for RADIUS type authentication mechanisms, and other types of authentication.

How easy is it to manage

Duo is managed through the Duo Security web portal. Your IT admins can manage users, MFA devices, tokens, and secured applications via the web interface. You can also deploy appliances that allow users to manage, provision, and add their MFA devices and settings.

Duo also integrates with Active Directory to make managing and maintaining users easy and fairly automated.

Let’s get started with Duo MFA

Want to protect your business with MFA? Give me a call today!

Jul 072020
 
Picture of a business office with cubicles

In the ever-evolving world of IT and End User Computing (EUC), new technologies and solutions are constantly being developed to decrease costs, improve functionality, and help the business’ bottom line. In this pursuit, as far as end user computing goes, two technologies have emerged: Hosted Desktop Infrastructure (HDI), and Virtual Desktop Infrastructure (VDI). In this post I hope to explain the differences and compare the technologies.

We’re at a point where due to the low cost of backend server computing, performance, and storage, it doesn’t make sense to waste end user hardware and resources. By deploying thin clients, zero clients, or software clients, we can reduce the cost per user for workstations or desktop computers, and consolidate these on the backend side of things. By moving moving EUC to the data center (or server room), we can reduce power requirements, reduce hardware and licensing costs, and take advantage of some cool technologies thanks to the use of virtualization and/or Storage (SANs), snapshots, fancy provisioning, backup and disaster recovery, and others.

See below for the video, or read on for the blog post!

And it doesn’t stop there, utilizing these technologies minimizes the resources required and spent on managing, monitoring, and supporting end user computing. For businesses this is a significant reduction in costs, as well as downtime.

What is Hosted Desktop Infrastructure (HDI) and Virtual Desktop Infrastructure (VDI)

Many IT professionals still don’t fully understand the difference between HDI and VDI, but it’s as sample as this: Hosted Desktop Infrastructure runs natively on the bare metal (whether it’s a server, or SoC) and is controlled and provided by a provisioning server or connection broker, whereas Virtual Desktop Infrastructure virtualizes (like you’re accustomed to with servers) the desktops in a virtual environment and is controlled and provided via hypervisors running on the physical hardware.

Hosted Desktop Infrastructure (HDI)

As mentioned above, Hosted Desktop Infrastructure hosts the End User Computing sessions on bare metal hardware in your datacenter (on servers). A connection broker handles the connections from the thin clients, zero clients, or software clients to the bare metal allowing the end user to see the video display, and interact with the workstation instance via keyboard and mouse.

Pros:

  • Remote Access capabilities
  • Reduction in EUC hardware and cost-savings
  • Simplifies IT Management and Support
  • Reduces downtime
  • Added redundancy
  • Runs on bare metal hardware
  • Resources are dedicated and not shared, the user has full access to the hardware the instance runs on (CPU, Memory, GPU, etc)
  • Easily provide accelerated graphics to EUC instances without additional costs
  • Reduction in licensing as virtualization products don’t need to be used

Cons:

  • Limited instance count to possible instances on hardware
  • Scaling out requires immediate purchase of hardware
  • Some virtualization features are not available since this solution doesn’t use virtualization
  • Additional backup strategy may need to be implemented separate from your virtualized infrastructure

Example:

If you require dedicated resources for end users and want to be as cost-effective as possible, HDI is a great candidate.

An example HDI deployment would utilize HPE Moonshot which is one of the main uses for HPE Moonshot 1500 chassis. HPE Moonshot allows you to provision up to 180 OS instances for each HPE Moonshot 1500 chassis.

More information on the HPE Moonshot (and HPE Edgeline EL4000 Converged Edge System) can be found here: https://www.stephenwagner.com/2018/08/22/hpe-moonshot-the-absolute-definition-of-high-density-software-defined-infrastructure/

Virtual Desktop Infrastructure (VDI)

Virtual Desktop Infrastructure virtualizes the end user operating system instances exactly how you virtualize your server infrastructure. In VMware environments, VMware Horizon View can provision, manage, and maintain the end user computing environments (virtual machines) to dynamically assign, distribute, manage, and broker sessions for users. The software product handles the connections and interaction between the virtualized workstation instances and the thin client, zero client, or software client.

Pros:

  • Remote Access capabilities
  • Reduction in EUC hardware and cost-savings
  • Simplifies IT Management and Support
  • Reduces downtime
  • Added redundancy
  • Runs as a virtual machine
  • Shared resources (you don’t waste hardware or resources as end users share the resources)
  • Easy to scale out (add more backend infrastructure as required, don’t need to “halt” scaling while waiting for equipment)
  • Can over-commit (over-provision)
  • Backup strategy is consistent with your virtualized infrastructure
  • Capabilities such as VMware DRS, VMware HA

Cons:

  • Resources are not dedicated and are shared, users share the server resources (CPU, Memory, GPU, etc)
  • Extra licensing may be required
  • Extra licensing required for virtual accelerated graphics (GPU)

Example:

If you want to share a pool of resources, require high availability, and/or have dynamic requirements then virtualization would be the way to go. You can over commit resources while expanding and growing your environment without any discontinuation of services. With virtualization you also have access to technologies such as DRS, HA, and special Backup and DR capabilities.

An example use case of VMware Horizon View and VDI can be found at: https://www.digitallyaccurate.com/blog/2018/01/23/vdi-use-case-scenario-machine-shops/

 Conclusion

Both technologies are great and have their own use cases depending on your business requirements. Make sure you research and weigh each of the options if you’re considering either technologies. Both are amazing technologies which will compliment and enhance your IT strategy.