Jul 162021
 

Well, it’s official, according to the release notes for VMware Horizon 2106, VMware now supports Media Optimization for Microsoft Teams on the VMware Horizon Linux Client.

This is great news for zero clients, as most VDI Zero Clients are based of embedded Linux. As soon as major vendors update their firmware to the latest VMware Horizon Client, we should start seeing Microsoft Teams Optimization on VDI Zero Clients.

To support this, you’ll need to have the proper configuration implemented. Make sure you check out my guide on Microsoft Teams VDI Optimization for VMware Horizon.

For the full release notes, click here.

May 152021
 
Image of an AMD S7150 X2 MxGPU GPU Graphics Card

The AMD S7150 x2 PCIe MxGPU is a Graphics card designed for multi-user (MxGPU) virtualized environments (VDI). Installing an AMD S7150 x2 MxGPU allows you to provision virtual GPUs to Virtual workstations to enable 3D acceleration for applications like engineering, gaming, or pretty much anything that requires accelerated graphics.

Being a big fan of VDI and having my own VDI homelab, I just had to get my hands on one of these cards to experiment with, and learn. It’s an older card that was released in February of 2016, but it’s perfect for the homelab enthusiast.

I secured one and here’s a story about how I got it working on an unsupported 1U HPE DL360p Gen8 Server.

AMD S7150 x2 Specifications

The S7150x2 features 2 physical GPUs, each with 8GB of Video RAM, while the little brother “S7150”, has one GPU and 8GB of Video RAM.

For cooling, the S7150x2 requires the server to cool the card (it has no active cooling or fans), whereas the S7150 is available as both active (with fan) cooling, and passive cooling.

This card supports older versions of VMware ESXi 6.5 and also some versions of Citrix XenServer.

AMD MxGPU Overview

A picture of an AMD S7150 x2 PCIe mxGPU Card
AMD S7150 x2 PCIe mxGPU Card

The AMD MxGPU technology, uses a technology called SR-IOV to create Virtual Functions (VFs) that can be attached to virtual machines.

The S7150 x2, with it’s 2GPUs can actually be carved up in to 32 (16 per GPU) VFs, providing 32 users with 3D accelerated graphics.

Additionally, you can simply passthrough the individual GPUs to VMs themselves without using SR-IOV and VFs, providing 2 users with vDGA PCIe Passthrough 3D Accelerated graphics. vDGA stands for “Virtual Dedicated Graphics Acceleration”.

Please Note: In order to use MxGPU capabilities, you must have a server that supports SR-IOV and be using a version of VMware that is compatible with the MxGPU drivers and configuration utility.

The AMD FirePro S7150 x2 does not have any video-out connectors or ports, this card is strictly designed to be used in virtual environments.

The AMD S7150 x2 connected to a HPE DL360p Gen8 Server

As most of you know, I maintain a homelab for training, learning, testing, and demo purposes. I’ve had the S7150 x2 for about 7 months or so, but haven’t been able to use it because I don’t have the proper server.

Securing the proper server is out of the question due to the expense as I fund the majority of my homelab myself, and no vendor has offered to provide me with a server yet (hint hint, nudge nudge).

I do have a HPE ML310e Gen8 v2 server that had an NVidia Grid K1 card which can physically fit and cool the S7150 x2, however it’s an entry-level server and there’s bugs and issues with PCIe passthrough. This means both vDGA and MxGPU are both out of the question.

Image of a AMD S7150 X2 side by side with an Nvidia GRID K1 GPU Graphics Card
AMD S7150 X2 side by side with an Nvidia GRID K1 GPU Graphics Card

All I have left are 2 x HPE DL360P Gen 8 Servers. They don’t fit double width PCIe cards, they aren’t on the supported list, and they can’t power the card, but HEY, I’m going to make this work!

Connecting the Card

To connect to the Server, I purchased a “LINKUP – 75cm PCIe 3.0 x16 Shielded PCI Express Extension Cable”. This is essentially just a really, very long PCIe extension ribbon cable.

I connected this to the inside of the server, gently folded the cable and fed it out the back of the server.

Picture of a Server with PCIe Extension Ribbon Cable to an external GPU
Server with PCIe Extension Ribbon Cable to an external GPU

I realized that when the cable came in contact with the metal frame, it actually peeled the rubber off the ribbon cable (very sharp), so be careful if you attempt this. Thankfully the cable is shielded and I didn’t cause any damage.

Cooling the Card

Cooling the card was one of the most difficult tasks. I couldn’t actually even test this card when I first purchased it, because after powering up a computer, the card would instantly get up to extremely hot temperatures. This forced me to power down the system before the OS even booted.

I purchased a couple 3D printed cooling kits off eBay, but unfortunately none worked as they were for Nvidia cards. Finally one day I randomly checked, and I finally found a 3D printed cooling solution specifically for the AMD S7150 x2.

Image of a AMD S7150 X2 Cooling Shroud and Fan
AMD S7150 X2 Cooling Shroud and Fan

As you can see, the kit included a 3D printed air baffle and a fan. I had to remove the metal holding bracket to install the air baffle.

I also had to purchase a PWM fan control module, as the fan included with the kit runs at 18,000 RPM. The exact item I purchased was a “Noctua NA-FC1, 4-Pin PWM Fan Controller”.

Image of an CFM Fan Control Module
CFM Fan Control Module

Once I installed the controller, I was able to run some tests adjusting the RPM while monitoring the temperatures of the card, and got the fan to a speed where it wasn’t audible, yet was able to cool and keep the GPUs between 40-51 degrees Celsius.

Powering the Card

The next problem I had to overcome was to power the card with it being external.

To do this, I purchased a Gigabyte P750GM Modular Power Supply. I chose this specific PSU because it’s modular and I only had to install the cables I required (being the 6-pin power cable, 8-pin power cable, ATX Power Cable (for PSU on switch), and a CFM fan power connector).

Picture of a Gigabyte P750GM Modular Power Supply (PSU)
Gigabyte P750GM Modular Power Supply (PSU)

As you can see in the picture below, I did not install all the cabling in the PSU.

Image of a Modular PSU Connected to AMD S7150 x2
Modular PSU Connected to AMD S7150 x2

As you can see, if came together quite nicely. I also had to purchase an ATX power on adapter, to short certain pins to power on the PSU.

Picture of ATX PSU Jump Adapter
ATX PSU Jump Adapter

I fed this cable under the PSU and it is hanging underneath the desk out of the way. Some day I might make my own adapter, so I can remove the ATX power connector but unfortunately the PIN-outs on the PSU don’t match the end of the ATX connector cable.

Image of Side view of external S7150 x2 GPU on Server
Side view of external S7150 x2 GPU on Server

It’s about as neat and tidy as it can be, being a hacked up solution.

Using the card

Overall, by the time I was done connecting it to the server, I was pretty happy with the cleaned up final result.

AMD S7150 x2 connected to HPE Proliant DL360p Gen8 Server
AMD S7150 x2 connected to HPE Proliant DL360p Gen8 Server

After booting the system, I noticed that VMware ESXi 6.5 detected the card and both GPUs.

Screenshot of AMD S7150 X2 PCIe Passthru ESXi 6.5
AMD S7150 X2 PCIe Passthru ESXi 6.5

You’ll notice that on the server, the GPUs show up as an “AMD Tonga S7150”.

Before I started to play around with the MxGPU software, I wanted to simply pass through an entire GPU to a VM for testing. I enabled ESXi Passthru on both GPUs, and restarted the server.

So far so good!

I already had a persistent VDI VM configured and ready to go, so I edited the VM properties, and attached one of the AMD S7150 x2 GPUs to the VM.

Screenshot of Attached S7150 x2 Tonga GPU to vSphere VDI VM PCIe Passthru
Attached S7150 x2 Tonga GPU to vSphere VDI VM PCIe Passthru

Booting the VM I was able to see the card and I installed the AMD Radeon FirePro drivers. Everything just worked! “dxdiag” was showing full 3D acceleration, and I confirmed that hardware h.264 offload with the VMware Horizon Agent was functioning (confirmed via BLAST session logs).

That was easy! 🙂

Issues

Now on to the issues. After spending numerous days, I was unable to get the MxGPU features working with the AMD Radeon FirePro drivers for VMware ESXi. However, thanks for a reader named TonyJr, I was able to solve this, but more on that later (keep reading).

Even though I had the drivers and the scripts installed, it was unable to create the VFs (Virtual Functions) with SR-IOV. From research on the internet with the limited amount of information there is, I came to believe that this is due to an SR-IOV bug on the Gen8 platform that I’m running (remember, this is completely and utterly NOT SUPPORTED).

If anyone is interested, the commands worked and the drivers loaded, but it just never created the functions on reboot. I also tried using the newer drivers for the V340 card, with no luck as the module wouldn’t even load.

Here is an example of the configuration script:

[root@DA-ESX03:/vmfs/volumes/5d40aefe-030ee1d6-df44-ecb1d7f30334/files/mxgpu] sh mxgpu-install.sh -c
Detected 2 SR-IOV GPU
0000:06:00.0 Display controller VGA compatible controller: AMD Tonga S7150 [vmgfx0]
0000:08:00.0 Display controller VGA compatible controller: AMD Tonga S7150 [vmgfx1]
Start configuration....
Do you plan to use the Radeon Pro Settings vSphere plugin to configure MxGPU? ([Y]es/[N]o, default:N)n
Default Mode
Enter the configuration mode([A]uto/[H]ybrid,default:A)a
Auto Mode Selected
Please enter number of VFs:(default:4): 2
Configuring the GPU 1 ...
0000:06:00.0 VGA compatible controller: AMD Tonga S7150 [vmgfx0]
GPU1=2,B6
Configuring the GPU 2 ...
0000:08:00.0 VGA compatible controller: AMD Tonga S7150 [vmgfx1]
GPU2=2,B8
Setting up SR-IOV settings...
Done
pciHole.start = 2048
pciHole.end = 4543
Eligible VMs:
DA-VDIWS01
DA-VDIWS02
DA-VDIUbuntu01
DA-MxGPU
PCI Hole settings will be added to these VMs. Is this OK?[Y/N]n
User Exit
The configuration needs a reboot to take effect

To automatically assign VFs, please run "sh mxgpu-install.sh -a" after system reboot
[root@DA-ESX03:/vmfs/volumes/5d40aefe-030ee1d6-df44-ecb1d7f30334/files/mxgpu]

And as mentioned, on reboot I would only be left with the actual 2 physical GPUs available for passthru.

I also tried using “esxcfg-module” utility to configure the driver, but that didn’t work either.

esxcfg-module -s "adapter1_conf=9,0,0,4,2048,4000" amdgpuv
esxcfg-module -s "adapter1_conf=9,0,0,2,4096,4000 adapter2_conf=11,0,0,2,4096,4000" amdgpuv

Both combinations failed to have any effect on creating the VFs. It was unfortunate, but I still had 2 separate GPUs that I could able to passthrough to 2 VDI VMs which is more than enough for me.

Issues (Update June 19 2022)

Thanks to “TonyJr” leaving a comment, I was able to get the MxGPU drivers functioning on the ESXi host.

To get SR-IOV and the drivers to function, I had to perform the following:

  1. Log on to the BIOS
  2. Press Ctrl+A which unlocked a secret Menu called “SERVICE OPTIONS”
  3. Open “SERVICE OPTIONS”
  4. Select “PCI Express 64-Bit BAR Support”, choose “Enable” and then reboot the server.

Upon reboot, the ESXi instance had actually already sliced up the S7150 MxGPU using the options I tried configuring above. It’s all working now!

Ultimately I tweaked the settings to only slice one of the two GPUs in to 2 VFs, leaving me with a full GPU for passthrough, as well as 2 VFs from the other GPU. Thanks TonyJr!

Horizon View with the S7150 x2

Right off the bat, I have to say this works AMAZING! I’ve been using this for about 4 weeks now without any issues (and no fires, lol).

As mentioned above, because of my issues with SR-IOV on the server I couldn’t utilize MxGPU, but I do have 2 full GPUs each with 8GB of VRAM each that I can passthrough to VDI Virtual Machines using vDGA. Let’s get in to the experience…

Similar to the experience with the Nvidia GRID K1 card, the S7150 x2 provides powerful 3D acceleration and GPU functionality to Windows VDI VMs. Animations, rendering, gaming, it all works and it’s all 3D accelerated!

I’ve even tested the S7150 x2 with my video editing software to edit and encode videos. No complaints and it works just like a desktop system with a high performance GPU would. Imagine video editing on the road with nothing but a cheap laptop and the VMware Horizon client software!

The card also offloads encoding of the VMware BLAST h.264 stream from the CPU to the GPU. This is what actually compresses the video display feed that goes from the VM to your VMware View client. This provides a smoother experience with no delay or lag, and frees up a ton of CPU cycles. Traditionally without a GPU to offload the encoding, the h.264 BLAST stream uses up a lot of CPU resources and bogs down the VDI VM (and the server it’s running on).

Unfortunately, I don’t have any engineering, mapping, or business applications to test with, that this card was actually designed for, but you have to remember this card was designed to provide VDI users with a powerful workstation experience.

It would be amazing if AMD (and other vendors) released more cards that could provide these capabilities, both for the enterprise as well as enthusiasts and their homelab.

May 142021
 

Welcome to Episode 02 of The Tech Journal Vlog at StephenWagner.com

In this episode

What I’ve done this week

  • 10ZiG Unboxing (10ZiG 4610q and 10ZiG 6110)
  • Thin Client Blogging and Video Creation
  • VDI Work (Instant Clones, NVME Flash Storage Server)

Fun Stuff

  • HPE Discover 2021 – June 22 to June 24 – Register for HPE Discover at https://infl.tv/jtHb
  • Firewall with 163 day uptime and no updates?!?!?
  • Microsoft Exchange Repeated Pending Reboot Issue
  • Microsoft Exchange Security Update KB5001779 (and CU18 to CU20)

Life Update

  • Earned VMware vExpert Status in February!
  • Starlink in Saskatchewan, Alberta (Canada)
    • VDI over Starlink, low latency!!!
    • Use Cases (Oil and Gas Facilities, etc)

Work Update

  • HPE Simplivity Upgrade (w/Identity Store Issues, Mellanox Firmware Issues)

New Blog Posts

Current Projects

  • 10ZiG 4610q Thin Client Content
  • 10ZiG 6110 Thin Client Content
  • VMware Horizon Instant Clones Guides and Content

Don’t forget to like and subscribe!
Leave a comment, feedback, or suggestions!

May 012021
 
Picture of NVMe Storage Server Project

For over a year and a half I have been working on building a custom NVMe Storage Server for my homelab. I wanted to build a high speed storage system similar to a NAS or SAN, backed with NVMe drives that provides iSCSI, NFS, and SMB Windows File Shares to my network.

The computers accessing the NVMe Storage Server would include VMware ESXi hosts, Raspberry Pi SBCs, and of course Windows Computers and Workstations.

The focus of this project is on high throughput (in the GB/sec) and IOPS.

The current plan for the storage environment is for video editing, as well as VDI VM storage. This can and will change as the project progresses.

The History

More and more businesses are using all-flash NVMe and SSD based storage systems, so I figured there’s no reason why I can’t have build and have my own budget custom all NVMe flash NAS.

This is the story of how I built my own NVMe based Storage Server.

The first version of the NVMe Storage Server consisted of the IO-PEX40152 card with 4 x 2TB Sabrent Rocket 4 NVMe drives inside of an HPE Proliant DL360p Gen8 Server. The server was running ESXi with TrueNAS virtualized, and the PCIe card passed through to the TrueNAS VM.

The results were great, the performance was amazing, and both servers had access to the NFS export via 2 x 10Gb SFP+ networking.

There were three main problems with this setup:

  1. Virtualized – Once a month I had an ESXi PSOD. This was either due to overheating of the IO-PEX40152 card because of modifications I made, or bugs with the DL360p servers and PCIe passthrough.
  2. NFS instead of iSCSI – Because TrueNAS was virtualized inside of the host that was using it for storage, I had to use NFS since the host virtualizing TrueNAS would also be accessing the data on the TrueNAS VM. When shutting down the host, you need to shut down TrueNAS first. NFS disconnects are handled way healthier than iSCSI disconnects (which can cause corruption even if no files are being used).
  3. CPU Cores maxed on data transfer – When doing initial testing, I was maxing out the CPU cores assigned to the TrueNAS VM because the data transfers were so high. I needed a CPU and setup that was better fit.

Version 1 went great, but you can see some things needed to be changed. I decided to go with a dedicated server, not virtualize TrueNAS, and go for a newer CPU with a higher Ghz speed.

And so, version 2 was born (built). Keep reading and scrolling for pictures!

The Hardware

On version 2 of the project, the hardware includes:

Notes on the Hardware:

  • While the ML310e Gen8 v2 server is a cheap low entry server, it’s been a fantastic team member of my homelab.
  • HPE Dual 10G Port 560SFP+ adapters can be found brand new in unsealed boxes on eBay at very attractive prices. Using HPE Parts inside of HPE Servers, avoids the fans from spinning up fast.
  • The ML310e Gen8 v2 has some issues with passing through PCIe cards to ESXi. Works perfect when not passing through.

The new NVMe Storage Server

I decided to repurpose an HPE Proliant ML310e Gen8 v2 Server. This server was originally acting as my Nvidia Grid K1 VDI server, because it supported large PCIe cards. With the addition of my new AMD S7150 x2 hacked in/on to one of my DL360p Gen8’s, I no longer needed the GRID card in this server and decided to repurpose it.

Picture of an HPe ML310e Gen8 v2 with NVMe Storage
HPe ML310e Gen8 v2 with NVMe Storage

I installed the IOCREST IO-PEX40152 card in to the PCIe 16x slot, with 4 x 2TB Sabrent Rocket 4 NVME drives.

Picture of IOCREST IO-PEX40152 with GLOTRENDS M.2 NVMe SSD Heatsink on Sabrent Rocket 4 NVME
IOCREST IO-PEX40152 with GLOTRENDS M.2 NVMe SSD Heatsink on Sabrent Rocket 4 NVME

While the server has a PCIe 16x wide slot, it only has an 8x bus going to the slot. This means we will have half the capable speed vs the true 16x slot. This however does not pose a problem because we’ll be maxing out the 10Gb NICs long before we max out the 8x bus speed.

I also installed an HPE Dual Port 560SFP+ NIC in to the second slot. This will allow a total of 2 x 10Gb network connections from the server to the Ubiquiti UniFi US-16-XG 10Gb network switch, the backbone of my network.

The Server also have 4 x Hot Swappable HD bays on the front. When configured in HBA mode (via the BIOS), these are accessible by TrueNAS and can be used. I plan on populating these with 4 x 4TB HPE MDL SATA Hot Swappable drives to act as a replication destination for the NVMe pool and/or slower magnetic long-term storage.

Front view of HPE ML310e Gen8 v2 with Hotswap Drive bays
HPE ML310e Gen8 v2 with Hotswap Drive bays

I may also try to give WD RED Pro drives a try, but I’m not sure if they will cause the fans to speed up on the server.

TrueNAS Installation and Configuration

For the initial Proof-Of-Concept for version 2, I decided to be quick and dirty and install it to a USB stick. I also waited until I installed TrueNAS on to the USB stick and completed basic configuration before installing the Quad NVMe PCIe card and 10Gb NIC. I’m using a USB 3.0 port on the back of the server for speed, as I can’t verify if the port on the motherboard is USB 2 or USB 3.

Picture of a TrueNAS USB Stick on HPE ML310e Gen8 v2
TrueNAS USB Stick on HPE ML310e Gen8 v2

TrueNAS installation worked without any problems whatsoever on the ML310e. I configured the basic IP, time, accounts, and other generic settings. I then proceeded to install the PCIe cards (storage and networking).

Screenshot of TrueNAS Dashboard Installed on NVMe Storage Server
TrueNAS Installed on NVMe Storage Server

All NVMe drives were recognized, along with the 2 HDDs I had in the front Hot-swap bays (sitting on an HP B120i Controller configured in HBA mode).

Screenshot of available TrueNAS NVMe Disks
TrueNAS NVMe Disks

The 560SFP+ NIC also was detected without any issues and available to configure.

Dashboard Screenshot of TrueNAS 560SFP+ 10Gb NIC
TrueNAS 560SFP+ 10Gb NIC

Storage Configuration

I’ve already done some testing and created a guide on FreeNAS and TrueNAS ZFS Optimizations and Considerations for SSD and NVMe, so I made sure to use what I learned in this version of the project.

I created a striped pool (no redundancy) of all 4 x 2TB NVMe drives. This gave us around 8TB of usable high speed NVMe storage. I also created some datasets and a zVOL for iSCSI.

Screenshot of NVMe TrueNAS Storage Pool with Datasets and zVol
NVMe TrueNAS Storage Pool with Datasets and zVol

I chose to go with the defaults for compression to start with. I will be testing throughput and achievable speeds in the future. You should always test this in every and all custom environments as the results will always vary.

Network Configuration

Initial configuration was done via the 1Gb NIC connection to my main LAN network. I had to change this as the 10Gb NIC will be directly connected to the network backbone and needs to access the LAN and Storage VLANs.

I went ahead and configured a VLAN Interface on VLAN 220 for the Storage network. Connections for iSCSI and NFS will be made on this network as all my ESXi servers have vmknics configured on this VLAN for storage. I also made sure to configure an MTU of 9000 for jumbo frames (packets) to increase performance. Remember that all hosts must have the same MTU to communicate.

Screenshot of 10Gb NIC on Storage VLAN
10Gb NIC on Storage VLAN

Next up, I had to create another VLAN interface for the LAN network. This would be used for management, as well as to provide Windows File Share (SMB/Samba) access to the workstations on the network. We leave the MTU on this adapter as 1500 since that’s what my LAN network is using.

Screenshot of 10Gb NIC on LAN VLAN
10Gb NIC on LAN VLAN

As a note, I had to delete the configuration for the existing management settings (don’t worry, it doesn’t take effect until you hit test) and configure the VLAN interface for my LANs VLAN and IP. I tested the settings, confirmed it was good, and it was all setup.

At this point, only the 10Gb NIC is now being used so I went ahead and disconnected the 1Gb network cable.

Sharing Setup and Configuration

It’s now time to configure the sharing protocols that will be used. As mentioned before, I plan on deploying iSCSI, NFS, and Windows File Shares (SMB/Samba).

iSCSI and NFS Configuration

Normally, for a VMware ESXi virtualization environment, I would always usually prefer iSCSI based storage, however I also wanted to configure NFS to test throughput of both with NVMe flash storage.

Earlier, I created the datasets for all my my NFS exports and a zVOL volume for iSCSI.

Note, that in order to take advantage of the VMware VAAI storage directives (enhancements), you must use a zVOL to present an iSCSI target to an ESXi host.

For NFS, you can simply create a dataset and then export it.

For iSCSI, you need to create a zVol and then configure the iSCSI Target settings and make it available.

SMB (Windows File Shares)

I needed to create a Windows File Share for file based storage from Windows computers. I plan on using the Windows File Share for high-speed storage of files for video editing.

Using the dataset I created earlier, I configured a Windows Share, user accounts, and tested accessing it. Works perfect!

Connecting the host

Connecting the ESXi hosts to the iSCSI targets and the NFS exports is done in the exact same way that you would with any other storage system, so I won’t be including details on that in this post.

We can clearly see the iSCSI target and NFS exports on the ESXi host.

Screenshot of TrueNAS NVMe iSCSI Target on VMware ESXi Host
TrueNAS NVMe iSCSI Target on VMware ESXi Host
Screenshot of NVMe iSCSI and NFS ESXi Datastores
NVMe iSCSI and NFS ESXi Datastores

To access Windows File Shares, we log on and map the network share like you would normally with any file server.

Testing

For testing, I moved (using Storage vMotion) my main VDI desktop to the new NVMe based iSCSI Target LUN on the NVMe Storage Server. After testing iSCSI, I then used Storage vMotion again to move it to the NFS datastore. Please see below for the NVMe storage server speed test results.

Speed Tests

Just to start off, I want to post a screenshot of a few previous benchmarks I compiled when testing and reviewing the Sabrent Rocket 4 NVMe SSD disks installed in my HPE DL360p Gen8 Server and passed through to a VM (Add NVMe capability to an HPE Proliant DL360p Gen8 Server).

Screenshot of CrystalDiskMark testing an IOCREST IO-PEX40152 and Sabrent Rocket 4 NVME SSD for speed
CrystalDiskMark testing an IOCREST IO-PEX40152 and Sabrent Rocket 4 NVME SSD
Screenshot of CrystalDiskMark testing IOPS on an IOCREST IO-PEX40152 and Sabrent Rocket 4 NVME SSD
CrystalDiskMark testing IOPS on an IOCREST IO-PEX40152 and Sabrent Rocket 4 NVME SSD

Note, that when I performed these tests, my CPU was maxed out and limiting the actual throughput. Even then, these are some fairly impressive speeds. Also, these tests were directly testing each NVMe drive individually.

Moving on to the NVMe Storage Server, I decided to test iSCSI NVMe throughput and NFS NVMe throughput.

I opened up CrystalDiskMark and started a generic test, running a 16GB test file a total of 6 times on my VDI VM sitting on the iSCSI NVMe LUN.

Screenshot of NVMe Storage Server iSCSI Benchmark with CrystalDiskMark
NVMe Storage Server iSCSI Benchmark with CrystalDiskMark

You can see some impressive speeds maxing out the 10Gb NIC with crazy performance of the NVME storage:

  • 1196MB/sec READ
  • 1145.28MB/sec WRITE (Maxing out the 10GB NIC)
  • 62,725.10 IOPS READ
  • 42,203.13 IOPS WRITE

Additionally, here’s a screenshot of the ix0 NIC on the TrueNAS system during the speed test benchmark: 1.12 GiB/s.

Screenshot of TrueNAS NVME Maxing out 10Gig NIC
TrueNAS NVME Maxing out 10Gig NIC

And remember this is with compression. I’m really excited to see how I can further tweak and optimize this, and also what increases will come with configuring iSCSI MPIO. I’m also going to try to increase the IOPS to get them closer to what each individual NVMe drive can do.

Now on to NFS, the results were horrible when moving the VM to the NFS Export.

Screenshot of NVMe Storage Server NFS Benchmark with CrystalDiskMark
NVMe Storage Server NFS Benchmark with CrystalDiskMark

You can see that the read speed was impressive, but the write speed was not. This is partly due to how writes are handled with NFS exports.

Clearly iSCSI is the best performing method for ESXi host connectivity to a TrueNAS based NVMe Storage Server. This works perfect because we’ll get the VAAI features (like being able to reclaim space).

iSCSI MPIO Speed Test

This is more of an update… I was finally able to connect, configure, and utilize the 2nd 10Gbe port on the 560SFP+ NIC. In my setup, both hosts and the TrueNAS storage server all have 2 connections to the switch, with 2 VLANs and 2 subnets dedicated to storage. Check out the before/after speed tests with enabling iSCSI MPIO.

As you can see I was able to essentially double my read speeds (again maxing out the networking layer), however you’ll notice that the write speeds maxed out at 1598MB/sec. I believe we’ve reached a limitation of the CPU, PCIe bus, or something else inside of the server. Note, that this is not a limitation of the Sabrent Rocket 4 NVME drives, or the IOCREST NVME PCIe card.

Moving Forward

I’ve had this configuration running for around a week now with absolutely no issues, no crashes, and it’s been very stable.

Using a VDI VM on NVMe backed storage is lightning fast and I love the experience.

I plan on running like this for a little while to continue to test the stability of the environment before making more changes and expanding the configuration and usage.

Future Plans (and Configuration)

  • Drive Bays
    • I plan to populate the 4 hot-swappable drive bays with HPE 4TB MDL drives. Configured with RaidZ1, this should give me around 12TB usable storage. I can use this for file storage, backups, replication, and more.
  • NVMe Replication
    • This design was focused on creating non-redundant extremely fast storage. Because I’m limited to a total of 4 NVMe disks in this design, I chose not to use RaidZ and striped the data. If one NVMe drive is lost, all data is lost.
    • I don’t plan on storing anything important, and at this point the storage is only being used for VDI VMs (which are backed up), and Video editing.
    • If I can populate the front drive bays, I can replicate the NVMe storage to the traditional HDD storage on a frequent basis to protect against failure to some level or degree.
  • Version 3 of the NVMe Storage Server
    • More NVMe and Bigger NVMe – I want more storage! I want to test different levels of RaidZ, and connect to the backbone at even faster speeds.
    • NVME Drives with PLP (Power Loss Prevention) for data security and protection.
    • Dual Power Supply

Let me know your thoughts and ideas on this setup!

Apr 252021
 
Screenshot of a Hybrid Azure AD Joined login

If you’re using Azure AD, and have Hybrid Azure AD joined machines, special considerations must be made with non-persistent VDI workstations and VMs. This applies to Instant Clones on VMware Horizon.

Due to the nature of non-persistent VDI, machines are created and destroyed on the fly with a user getting an entirely new workstation on every login.

Hybrid Azure AD joined workstations not only register on the local domain Active Directory, but also register on the Azure AD (Azure Active Directory).

The Problem

If you have Hybrid Azure AD configured and machines performing the Hybrid Join, this will cause numerous machines to be created on Azure AD, in a misconfigured and/or unregistered state. When the non-persistent instant clone is destroyed and re-created, it will potentially have the same computer name as a previous machine, but will be unable to utilize the existing registration.

This conflict state could potentially make your Azure AD computer OU a mess.

VMware Horizon 8 version 2303 now supports Hybrid Azure AD joined non-persistent instant clones using Azure AD Connect. If you are using an older version, or using a different platform for non-persistent VDI, you’ll need to reference the solution below.

The Solution

Please see below for a few workarounds and/or solutions:

  1. Upgrade to VMware Horizon 8 2303
  2. Use Seamless SSO instead of Hybrid Azure AD join (click here for more information)
  3. Utilize login/logoff scripts to Azure AD join and unjoin on user login/logoff. You may have to create a cleanup script to remove old/stale records from Azure AD as this can and will create numerous computer accounts on Azure AD.
  4. Do not allow non-persistent virtual machines to Hybrid Domain Join. This can be accomplished either by removing the non-persistent VDI computer OU from synchronization with Azure AD Connect (OU Filtering information at https://docs.microsoft.com/en-us/azure/active-directory/hybrid/how-to-connect-sync-configure-filtering) or by disabling the scheduled task to perform an Azure AD join.

In my environment I elected to remove the non-persistent computer OU from Azure AD Connect sync, and it’s been working great. It also keeps my Azure Active Directory nice and clean.