Oct 232020
 
vCSA Update Installation

When updating VMware vCenter vCSA 7.0 U1 (Build 16858589) to vCSA 7.0 U1 (Build 17004997/17005016, Version 7.0.1.00100), you may notice that the update fails and reports issues with pre-update checks.

Pre-update checks done prior to the update will pass and allow you to proceed, however it’s the installation that will fail and crash reporting this error.

After the installation fails, you will no longer be able to log in to the vCSA VAMI reporting the error “Unable to Login” using the root account.

You are able to login via SSH. Resetting the root password via SSH will not resolve this issue.

The Problem

In the past, issues with the root password expiring have caused similar behavior on the vCSA VAMI. Changing the root password does not resolve this specific issue.

Further troubleshooting, it appears that special characters in the root password such as “!”, “.”, and “@” caused this issue to occur in my environment.

I was not able to fix the broken vCSA after the failed update. Access to the vCSA was not possible, however vCenter functions were still operating.

The Solution

To resolve this situation in my environment, I restored a snapshot of the vCSA taken prior to updating.

After restoring the snapshot, I changed the root password for VAMI and restarted the vCSA.

Another snapshot was taken prior to attempting the upgrade, which was now succesfull after removing special characters out of the root password.

Oct 152020
 
VMware vCLS VM in VM List

Did a new VM appear on your VMware vSphere cluster called “vCLS”? Maybe multiple appeared named “vCLS (1)”, “vCLS (2)”, “vCLS (3)” appeared.

VMware vCLS VM in vSphere Cluster Objects
VMware vCLS VM in vSphere Cluster Objects

This could be frightening but fear not, this is part of VMware vSphere 7.0 Update 1.

What is the vCLS VM?

The vCLS virtural machine is essentially an “appliance” or “service” VM that allows a vSphere cluster to remain functioning in the event that the vCenter Server becomes unavailable. It will maintain the health and services of that cluster.

Where did the vCLS VM come from?

The vCLS VM will appear after upgrading to vSphere 7.0 Update 1. I’m assuming it was deployed during the upgrade process.

It does not appear in the standard Cluster, Hosts, and VMs view, but does appear when looking at the vSphere objects VM lists, Storage VM lists, etc…

Is it normal to have more than one vCLS VM?

The vCLS VMs are created when hosts are added to a vSphere Cluster. Up to 3 vCLS VMs are required to run in each vSphere Cluster.

The vCLS VMs will also appear on clusters which contain only one or two hosts. These configurations will result in either 1 or 2 vCLS VMs named “vCLS (1)” and “vCLS (2)”.

More Information on vCLS VMs

For more information and technical specifics, you can visit the link below:

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenterhost.doc/GUID-96BD6016-4BE7-4B1C-8269-568D1555B08C.html

Hope this posts helps, and puts some minds at ease. Your VMware environment has NOT been compromised.

Oct 102020
 

If you’re like me and use an older Nvidia GRID K1 or K2 vGPU video card for your VDI homelab, you may notice that when using VMware Horizon that VMware Blast h264 encoding is no longer being offloaded to the GPU and is instead being encoded via the CPU.

The Problem

Originally when an environment was configured with an Nvidia GRID K1 or K2 card, not only does the card provide 3D acceleration and rendering, but it also offloads the VMware BLAST h264 stream (the visual session) so that the CPU doesn’t have to. This results in less CPU usage and provides a streamlined experience for the user.

This functionality was handled via NVFBC (Nvidia Frame Buffer Capture) which was part of the Nvidia Capture SDK (formerly known as GRID SDK). This function allowed the video card to capture the video frame buffer and encode it using NVENC (Nvidia Encoder).

Ultimately after spending hours troubleshooting, I learned that NVFBC has been deprecated and is no longer support, hence why it’s no longer functioning. I also checked and noticed that tools (such as nvfbcenable) were no longer bundled with the VMware Horizon agent. One can assume that the agent doesn’t even attempt to check or use this function.

Symptoms

Before I was aware of this, I noticed that while 3D Acceleration and graphics were functioning, I was experiencing high CPU usage. Upon further investigation I noticed that my VMware BLAST sessions were not offloading h264 encoding to the video card.

VMware Horizon Performance Tracker
VMware Horizon Performance Tracker with NVidia GRID K1

You’ll notice above that under the “Encoder” section, the “Encoder Name” was listed as “h264 4:2:0”. Normally this would say “NVIDIA NvEnc H264” (or “NVIDIA NvEnc HEVC” on newer cards) if it was being offloaded to the GPU.

Looking at a VMware Blast session (Blast-Worker-SessionId1.log), the following lines can be seen.

[INFO ] 0x1f34 bora::Log: NvEnc: VNCEncodeRegionNvEncLoadLibrary: Loaded NVIDIA SDK shared library "nvEncodeAPI64.dll"
[INFO ] 0x1f34 bora::Log: NvEnc: VNCEncodeRegionNvEncLoadLibrary: Loaded NVIDIA SDK shared library "nvml.dll"
[WARN ] 0x1f34 bora::Warning: GetProcAddress: Failed to resolve nvmlDeviceGetEncoderCapacity: 127
[WARN ] 0x1f34 bora::Warning: GetProcAddress: Failed to resolve nvmlDeviceGetProcessUtilization: 127
[WARN ] 0x1f34 bora::Warning: GetProcAddress: Failed to resolve nvmlDeviceGetGridLicensableFeatures: 127
[INFO ] 0x1f34 bora::Log: NvEnc: VNCEncodeRegionNvEncLoadLibrary: Some NVIDIA nvml functions unavailable, unloading
[INFO ] 0x1f34 bora::Log: NvEnc: VNCEncodeRegionNvEncUnloadLibrary: Unloading NVIDIA SDK shared library "nvEncodeAPI64.dll"
[INFO ] 0x1f34 bora::Log: NvEnc: VNCEncodeRegionNvEncUnloadLibrary: Unloading NVIDIA SDK shared library "nvml.dll"
[WARN ] 0x1f34 bora::Warning: GetProcAddress: Failed to resolve nvmlDeviceGetEncoderCapacity: 127
[WARN ] 0x1f34 bora::Warning: GetProcAddress: Failed to resolve nvmlDeviceGetProcessUtilization: 127
[WARN ] 0x1f34 bora::Warning: GetProcAddress: Failed to resolve nvmlDeviceGetGridLicensableFeatures: 127

You’ll notice it tries to load the proper functions, however it fails.

The Solution

Unfortunately the only solution is to upgrade to newer or different hardware.

The GRID K1 and GRID K2 cards have reached their EOL (End of Life) and are no longer support. The drivers are not being maintained or updated so I doubt they will take advantage of the newer frame buffer capture functions of Windows 10.

Newer hardware and solutions have incorporated this change and use a different means of frame buffer capture.

To resolve this in my own homelab, I plan to migrate to an AMD FirePro S7150x2.

Jul 222020
 
VMware Logon

When upgrading from any version of VMware vCSA to version 7.0, you may encounter a problem during the migration phase and be asked to specifiy a new “Export Directory”.

I’ve seen this occur on numerous upgrades and often find the same culprit causing the issue. I’ve found a very simple fix compared to other solutions online.

The full prompt for this issue is: “Enter a new export directory on the source machine below”

The Problem

When you upgrade the vCenter vCSA, the process migrates all data over from the source appliance, to the new vCSA 7 appliance.

This data can include the following (depending on your selection):

  • Configuration
  • Configuration and historical data (events and tasks)
  • Configuration and historical data (events, tasks, and performance metric)
  • vSphere Update Manager (updates, configuration, etc.)

This data can accumulate, especially the VMware vSphere Update Manager.

In the most recent upgrade I performed, I noticed that the smallest option (configuration only) was around 8GB, which is way over the 4.7GB default limit.

Could it be vSphere Update Manager?

I’ve seen VMware VUM cause numerous issues over the years with upgrades. VUM has caused issues upgrading from earlier versions to 6.x, and in this case it caused this issue upgrading to vCSA 7.x as well.

In my diagnosis, I logged in to the SSH console of the source appliance, and noticed that the partitions containing the VUM data (which includes update files) was around 7.4GB. This is the “/storage/updatemgr/” partition.

I wasn’t sure if this was included, but the 8GB of configuration, minus the 7.4GB of VUM data, could technically get me to around 0.6GB for migration if this was in fact included.

In my environment, I have the default (and simple) implementation of VUM with the only customization being the HPE VIBs depot. I figured maybe I should blast away the VUM and start from scratch on VMware vCSA 7.0 to see if this fixes the issue.

The Fix

To fix this issue, I simply completely reset the VMware Update Manager Database.

For details on this process and before performing these steps, please see VMware KB 2147284.

Let’s get to it:

  1. Close the migration window (you can reopen this later)
  2. Log in to your vCSA source appliance via SSH or console
  3. Run the applicable steps as defined in the VMware KB 2147284 to reset VUM (WARNING: commands are version specific). In my case on vCSA 6.5 I ran the following commands:
    1. shell
    2. service-control --stop vmware-updatemgr
    3. /usr/lib/vmware-updatemgr/bin/updatemgr-util reset-db
    4. rm -rf /storage/updatemgr/patch-store/*
    5. service-control --start vmware-updatemgr
  4. Open your web broswer and navigate to https://new-vcsa-IP:5480 and resume the migration. You will now notice a significant space reduction and won’t need to specify a new mount point.

That’s it! You have a shiny new clean VUM instance, and can successfully upgrade to vCSA 7.0 without having to specify a new mount point.

To restore any old configuration to VUM, you’ll do so in the “VMware Lifecycle Management” section of the VMware vCenter Server Appliance interface.

Jul 132020
 
Picture of the DUO Security Logo

When you’re looking for additional or enhanced options to secure you’re business and enterprise IT systems, MFA/2FA can help you achieve this. Get away from the traditional single password, and implement additional means of authentication! MFA provides a great compliment to your cyber-security policies.

My company, Digitally Accurate Inc, has been using the Duo Security‘s MFA product in our own infrastructure, as well as our customers environments for some time. Digitally Accurate is a DUO Partner and can provide DUO MFA Services including licensing/software and the hardware tokens (Duo D-100 Tokens using HOTP).

What is MFA/2FA

MFA is short for Multi Factor authentication, additionally 2FA is short for Two Factor Authentication. While they are somewhat the same, multi means many, and 2 means two. Additional security is provided with both, since it provides more means of authentication.

Traditionally, users authenticate with 1 (one) level of authentication: their password. In simple terms MFA/2FA in addition to a password, provides a 2nd method of authentication and identity validation. By requiring users to authentication with a 2nd mechanism, this provides enhanced security.

Why use MFA/2FA

In a large portion of security breaches, we see users passwords become compromised. This can happen during a phishing attack, virus, keylogger, or other ways. Once a malicious user or bot has a users credentials (username and password), they can access resources available to that user.

By implementing a 2nd level of authentication, even if a users password becomes compromised, the real (or malicious user) must pass a 2nd authentication check. While this is easy for the real user, in most cases it’s nearly impossible for a malicious user. If a password get’s compromised, nothing can be accessed as it requires a 2nd level of authentication. If this 2nd method is a cell phone or hardware token, a malicious user won’t be ale to access the users resources unless they steal the cell phone, or hardware token.

How does MFA/2FA work

When deploying MFA or 2FA you have the option of using an app, hardware token (fob), or phone verification to perform the additional authentication check.

After a user attempts to logs on to a computer or service with their username and password, the 2nd level of authentication will be presented, and must pass in order for the login request to succeed.

Please see below for an example of 2FA selection screen after a successful username and password:

Screenshot of Duo MFA 2FA Prompt on Windows Login
Duo Security Windows Login MFA 2FA Prompt

After selecting an authentication method for MFA or 2FA, you can use the following

2FA with App (Duo Push)

Duo Push sends an authentication challenge to your mobile device which a user can then approve or deny.

Please see below for an example of Duo Push:

Screenshot of Duo Push Notification to Mobile Android App
Duo Push to Mobile App on Android

Once the user selects to approve or deny the login request, the original login will either be approved or denied. We often see this as being the preferred MFA/2FA method.

2FA with phone verification (Call Me)

Duo phone verification (Call Me) will call you on your phone number (pre-configured by your IT staff) and challenge you to either hangup to deny the login request, or press a button on the keypad to accept the login request.

While we rarely use this option, it is handy to have as a backup method.

2FA with Hardware Token (Passcode)

Duo Passcode challenges are handled using a hardware token (or you can generate a passcode using the Duo App). Once you select this method, you will be prompted to enter the passcode to complete the 2FA authentication challenge. If you enter the correct passcode, the login will be accepted.

Here is a Duo D-100 Token that uses HOTP (HMAC-based One Time Password):

Picture of Duo D-100 HOTP Hardware Token
Duo D-100 HOTP Hardware Token

When you press the green button, a passcode will be temporarily displayed on the LCD display which you can use to complete the passcode challenge.

You can purchase Hardware Token’s directly from Digitally Accurate Inc by contacting us, your existing Duo Partner, or from Duo directly. Duo is also compatible with other 3rd party hardware tokens that use HOTP and TOTP.

2FA with U2F

While you can’t visibly see the option for U2F, you can use U2F as an MFA or 2FA authentication challenge. This includes devices like a Yubikey from Yubico, which plugs in to the USB port of your computer. You can attach a Yubikey to your key chain, and bring it around with you. The Yubikey simply plugs in to your USB port and has a button that you press when you want to authenticate.

When the 2FA window pops up, simply hit the button and your Yubikey will complete the MFA/2FA challange.

What can MFA/2FA protect

Duo MFA supports numerous cloud and on-premise applications, services, protocols, and technologies. While the list is very large (full list available at https://duo.com/product/every-application), we regularly deploy and use Duo Security for the following configurations.

Windows Logins (Server and Workstation Logon)

Duo MFA can be deployed to not only protect your Windows Servers and Workstations, but also your remote access system as well.

When logging on to a Windows Server or Windows Workstation, a user will be presented with the following screen for 2FA authentication:

Screenshot of Duo MFA 2FA Prompt on Windows Login
Duo Security Windows Login MFA 2FA Prompt

Below you can see a video demonstration of DUO on Windows Login.

DUO works with both Windows Logins and RDP (Remote Desktop Protocol) Logins.

VMWare Horizon View Clients (VMWare VDI Logon)

Duo MFA can be deployed to protect your VDI (Virtual Desktop Infrastructure) by requiring MFA or 2FA when users log in to access their desktops.

When logging on to the VMware Horizon Client, a user will be presented with the following screen for 2FA authentication:

Screenshot of Duo MFA 2FA Prompt on VMWare Horizon Client Login
Duo Security VMWare Horizon Client Login MFA 2FA Prompt

Below you can see a video demonstration of DUO on VMware Horizon View (VDI) Login.

Sophos UTM (Admin and User Portal Logon)

Duo MFA can be deployed to protect your Sophos UTM firewall. You can protect the admin account, as well as user accounts when accessing the user portal.

If you’re using the VPN functionality on the Sophos UTM, you can also protect VPN logins with Duo MFA.

Unix and Linux (Server and Workstation Logon)

Duo MFA can be deployed to protect your Unix and Linux Servers. You can protect all user accounts, including the root user.

We regularly deploy this with Fedora and CentOS (even FreePBX) and you can protect both SSH and/or console logins.

When logging on to a Unix or Linux server, a user will be presented with the following screen for 2FA authentication:

Screenshot of Duo MFA 2FA Prompt on CentOS Linux Login
Duo Security CentOS Linux login MFA 2FA Prompt

Below you can see a video demonstration of DUO on Linux.

WordPress Logon

Duo MFA can be deployed to protect your WordPress blog. You can protect your admin and other user accounts.

If you have a popular blog, you know how often bots are attempting to hack and brute force your passwords. If by chance your admin password becomes compromised, using MFA or 2FA can protect your site.

When logging on to a WordPress blog admin interface, a user will be presented with the following screen for 2FA authentication:

Screenshot of Duo MFA 2FA Prompt on WordPress Login
Duo Security WordPress Login MFA 2FA Prompt

Below you can see a video demonstration of DUO on a WordPress blog.

How easy is it to implement

Implementing Duo MFA is very easy and works with your existing IT Infrastructure. It can easily be setup, configured, and maintained on your existing servers, workstations, and network devices.

Duo offers numerous plugins (for windows), as well as options for RADIUS type authentication mechanisms, and other types of authentication.

How easy is it to manage

Duo is managed through the Duo Security web portal. Your IT admins can manage users, MFA devices, tokens, and secured applications via the web interface. You can also deploy appliances that allow users to manage, provision, and add their MFA devices and settings.

Duo also integrates with Active Directory to make managing and maintaining users easy and fairly automated.

Let’s get started with Duo MFA

Want to protect your business with MFA? Give me a call today!

Jul 072020
 
Picture of a business office with cubicles

In the ever-evolving world of IT and End User Computing (EUC), new technologies and solutions are constantly being developed to decrease costs, improve functionality, and help the business’ bottom line. In this pursuit, as far as end user computing goes, two technologies have emerged: Hosted Desktop Infrastructure (HDI), and Virtual Desktop Infrastructure (VDI). In this post I hope to explain the differences and compare the technologies.

We’re at a point where due to the low cost of backend server computing, performance, and storage, it doesn’t make sense to waste end user hardware and resources. By deploying thin clients, zero clients, or software clients, we can reduce the cost per user for workstations or desktop computers, and consolidate these on the backend side of things. By moving moving EUC to the data center (or server room), we can reduce power requirements, reduce hardware and licensing costs, and take advantage of some cool technologies thanks to the use of virtualization and/or Storage (SANs), snapshots, fancy provisioning, backup and disaster recovery, and others.

And it doesn’t stop there, utilizing these technologies minimizes the resources required and spent on managing, monitoring, and supporting end user computing. For businesses this is a significant reduction in costs, as well as downtime.

What is Hosted Desktop Infrastructure (HDI) and Virtual Desktop Infrastructure (VDI)

Many IT professionals still don’t fully understand the difference between HDI and VDI, but it’s as sample as this: Hosted Desktop Infrastructure runs natively on the bare metal (whether it’s a server, or SoC) and is controlled and provided by a provisioning server or connection broker, whereas Virtual Desktop Infrastructure virtualizes (like you’re accustomed to with servers) the desktops in a virtual environment and is controlled and provided via hypervisors running on the physical hardware.

Hosted Desktop Infrastructure (HDI)

As mentioned above, Hosted Desktop Infrastructure hosts the End User Computing sessions on bare metal hardware in your datacenter (on servers). A connection broker handles the connections from the thin clients, zero clients, or software clients to the bare metal allowing the end user to see the video display, and interact with the workstation instance via keyboard and mouse.

Pros:

  • Remote Access capabilities
  • Reduction in EUC hardware and cost-savings
  • Simplifies IT Management and Support
  • Reduces downtime
  • Added redundancy
  • Runs on bare metal hardware
  • Resources are dedicated and not shared, the user has full access to the hardware the instance runs on (CPU, Memory, GPU, etc)
  • Easily provide accelerated graphics to EUC instances without additional costs
  • Reduction in licensing as virtualization products don’t need to be used

Cons:

  • Limited instance count to possible instances on hardware
  • Scaling out requires immediate purchase of hardware
  • Some virtualization features are not available since this solution doesn’t use virtualization
  • Additional backup strategy may need to be implemented separate from your virtualized infrastructure

Example:

If you require dedicated resources for end users and want to be as cost-effective as possible, HDI is a great candidate.

An example HDI deployment would utilize HPE Moonshot which is one of the main uses for HPE Moonshot 1500 chassis. HPE Moonshot allows you to provision up to 180 OS instances for each HPE Moonshot 1500 chassis.

More information on the HPE Moonshot (and HPE Edgeline EL4000 Converged Edge System) can be found here: https://www.stephenwagner.com/2018/08/22/hpe-moonshot-the-absolute-definition-of-high-density-software-defined-infrastructure/

Virtual Desktop Infrastructure (VDI)

Virtual Desktop Infrastructure virtualizes the end user operating system instances exactly how you virtualize your server infrastructure. In VMware environments, VMware Horizon View can provision, manage, and maintain the end user computing environments (virtual machines) to dynamically assign, distribute, manage, and broker sessions for users. The software product handles the connections and interaction between the virtualized workstation instances and the thin client, zero client, or software client.

Pros:

  • Remote Access capabilities
  • Reduction in EUC hardware and cost-savings
  • Simplifies IT Management and Support
  • Reduces downtime
  • Added redundancy
  • Runs as a virtual machine
  • Shared resources (you don’t waste hardware or resources as end users share the resources)
  • Easy to scale out (add more backend infrastructure as required, don’t need to “halt” scaling while waiting for equipment)
  • Can over-commit (over-provision)
  • Backup strategy is consistent with your virtualized infrastructure
  • Capabilities such as VMware DRS, VMware HA

Cons:

  • Resources are not dedicated and are shared, users share the server resources (CPU, Memory, GPU, etc)
  • Extra licensing may be required
  • Extra licensing required for virtual accelerated graphics (GPU)

Example:

If you want to share a pool of resources, require high availability, and/or have dynamic requirements then virtualization would be the way to go. You can over commit resources while expanding and growing your environment without any discontinuation of services. With virtualization you also have access to technologies such as DRS, HA, and special Backup and DR capabilities.

An example use case of VMware Horizon View and VDI can be found at: https://www.digitallyaccurate.com/blog/2018/01/23/vdi-use-case-scenario-machine-shops/

 Conclusion

Both technologies are great and have their own use cases depending on your business requirements. Make sure you research and weigh each of the options if you’re considering either technologies. Both are amazing technologies which will compliment and enhance your IT strategy.

May 262020
 

So you want to add NVMe storage capability to your HPE Proliant DL360p Gen8 (or other Proliant Gen8 server) and don’t know where to start? Well, I was in the same situation until recently. However, after much research, a little bit of spending, I now have 8TB of NVMe storage in my HPE DL360p Gen8 Server thanks to the IOCREST IO-PEX40152.

Unsupported you say? Well, there are some of us who like to live life dangerously, there is also those of us with really cool homelabs. I like to think I’m the latter.

PLEASE NOTE: This is not a supported configuration. You’re doing this at your own risk. Also, note that consumer/prosumer NVME SSDs do not have PLP (Power Loss Prevention) technology. You should always use supported configurations and enterprise grade NVME SSDs in production environments.

DISCLAIMER: If you attempt what I did in this post, you are doing it at your own risk. I won’t be held liable for any damages or issues.

Use Cases

There’s a number of reasons why you’d want to do this. Some of them include:

  • Server Storage
  • VMware Storage
  • VMware vSAN
  • Virtualized Storage (SDS as example)
  • VDI
  • Flash Cache
  • Special applications (database, high IO)

Adding NVMe capability

Well, after all that research I mentioned at the beginning of the post, I installed an IOCREST IO-PEX40152 inside of an HPE Proliant DL360p Gen8 to add NVMe capabilities to the server.

IOCREST IO-PEX40152 with 4 x 2TB Sabrent Rocket 4 NVME

At first I was concerned about dimensions as technically the card did fit, but technically it didn’t. I bought it anyways, along with 4 X 2TB Sabrent Rocket 4 NVMe SSDs.

The end result?

Picture of an HPE DL360p Gen8 with NVME SSD
HPE DL360p Gen8 with NVME SSD

IMPORTANT: Due to the airflow of the server, I highly recommend disconnecting and removing the fan built in to the IO-PEX40152. The DL360p server will create more than enough airflow and could cause the fan to spin up, generate electricity, and damage the card and NVME SSD.

Also, do not attempt to install the case cover, additional modification is required (see below).

The Fit

Installing the card inside of the PCIe riser was easy, but snug. The metal heatsink actually comes in to contact with the metal on the PCIe riser.

Picture of an IO-PEX40152 installed on DL360p PCIe Riser
IO-PEX40152 installed on DL360p PCIe Riser

You’ll notice how the card just barely fits inside of the 1U server. Some effort needs to be put in to get it installed properly.

Picture of an DL360p Gen8 1U Rack Server with IO-PEX40152 Installed
HPE DL360p Gen8 with IO-PEX40152 Installed

There are ribbon cables (and plastic fittings) directly where the end of the card goes, so you need to gently push these down and push cables to the side where there’s a small amount of thin room available.

We can’t put the case back on… Yet!

Unfortunately, just when I thought I was in the clear, I realized the case of the server cannot be installed. The metal bracket and locking mechanism on the case cover needs the space where a portion of the heatsink goes. Attempting to install this will cause it to hit the card.

Picture of the HPE DL360p Gen8 Case Locking Mechanism
HPE DL360p Gen8 Case Locking Mechanism

The above photo shows the locking mechanism protruding out of the case cover. This will hit the card (with the IOCREST IO-PEX40152 heatsink installed). If the heatsink is removed, the case might gently touch the card in it’s unlocked and recessed position, but from my measurements clears the card when locked fully and fully closed.

I had to come up with a temporary fix while I figure out what to do. Flip the lid and weight it down.

Picture of an HPE DL360p Gen8 case cover upside down
HPE DL360p Gen8 case cover upside down

For stability and other tests, I simply put the case cover on upside down and weighed it down with weights. Cooling is working great and even under high load I haven’t seen the SSD’s go above 38 Celsius.

The plan moving forward was to remove the IO-PEX40152 heatsink, and install individual heatsinks on the NVME SSD as well as the PEX PCIe switch chip. This should clear up enough room for the case cover to be installed properly.

The fix

I went on to Amazon and purchased the following items:

4 x GLOTRENDS M.2 NVMe SSD Heatsink for 2280 M.2 SSD

1 x BNTECHGO 4 Pcs 40mm x 40mm x 11mm Black Aluminum Heat Sink Cooling Fin

They arrived within days with Amazon Prime. I started to install them.

Picture of Installing GLOTRENDS M.2 NVMe SSD Heatsink on Sabrent Rocket 4 NVME
Installing GLOTRENDS M.2 NVMe SSD Heatsink on Sabrent Rocket 4 NVME
Picture of IOCREST IO-PEX40152 with GLOTRENDS M.2 NVMe SSD Heatsink on Sabrent Rocket 4 NVME
IOCREST IO-PEX40152 with GLOTRENDS M.2 NVMe SSD Heatsink on Sabrent Rocket 4 NVME

And now we install it in the DL360p Gen8 PCIe riser and install it in to the server.

You’ll notice it’s a nice fit! I had to compress some of the heat conductive goo on the PFX chip heatsink as the heatsink was slightly too high by 1/16th of an inch. After doing this it fit nicely.

Also, note the one of the cable/ribbon connectors by the SAS connections. I re-routed on of the cables between the SAS connectors they could be folded and lay under the card instead of pushing straight up in to the end of the card.

As I mentioned above, the locking mechanism on the case cover may come in to contact with the bottom of the IOCREST card when it’s in the unlocked and recessed position. With this setup, do not unlock the case or open the case when the server is running/plugged in as it may short the board. I have confirmed when it’s closed and locked, it clears the card. To avoid “accidents” I may come up with a non-conductive cover for the chips it hits (to the left of the fan connector on the card in the image).

And with that, we’ve closed the case on this project…

Picture of a HPE DL360p Gen8 Case Closed
HPE DL360p Gen8 Case Closed

One interesting thing to note is that the NVME SSD are running around 4-6 Celsius cooler post-modification with custom heatsinks than with the stock heatsink. I believe this is due to the awesome airflow achieved in the Proliant DL360 servers.

Conclusion

I’ve been running this configuration for 6 days now stress-testing and it’s been working great. With the server running VMware ESXi 6.5 U3, I am able to passthrough the individual NVME SSD to virtual machines. Best of all, installing this card did not cause the fans to spin up which is often the case when using non-HPE PCIe cards.

This is the perfect mod to add NVME storage to your server, or even try out technology like VMware vSAN. I have a number of cool projects coming up using this that I’m excited to share.

May 252020
 
vSphere Logo Image

When troubleshooting connectivity issues with your vMotion network (or vMotion VLAN), you may notice that you’re unable to ping using the ping or vmkping command on your ESXi and VMware hosts.

This occurs when you’re suing the vMotion TCP/IP stack on your vmkernel (vmk) adapters that are configured for vMotion.

This also applies if you’re using long distance vMotion (LDVM).

Why

The vMotion TCP/IP stack requires special syntax for ping and ICMP tests on the vmk adapters.

A screenshot of vmk adapters, one of which is using the vMotion TCP/IP Stack
VMK using vMotion TCP/IP Stack

Above is an example where a vmk adapter (vmk3) is configured to use the vMotion TCP/IP stack.

How

To “ping” and test your vMotion network that uses the vMotion TCP/IP stack, you’ll need to use the special command below:

esxcli network diag ping -I vmk1 --netstack=vmotion -H ip.add.re.ss

In the command above, change “vmk1” to the vmkernel adapter you want to send the pings from. Additionally, change “ip.add.re.ss” to the IP address of the host you want to ping.

Using this method, you can fully verify network connectivity between the vMotion vmks using the vMotion stack.

Additional information and examples can be found at https://kb.vmware.com/s/article/59590.

Apr 072020
 
VMware Horizon View Icon

In response to COVID 19, VMware has extended their VMware Horizon 7 trial offering up to 90 days and includes 100 users. This includes both VMware Horizon 7 On-Premise, as well as VMware Cloud on AWS.

This is great if you’re planning or about to implement and deploy VMware Horizon 7.

In it’s simplest form, Horizon 7 allows an organization to virtualize their end user computing. No more computers, no more desktops, only Zero clients and software clients. Not only does this streamline the end user computing experience, but it enables a beautiful remote access solution as well.

And Horizon isn’t limited to VDI… You can install the VMware Horizon Agent on a Physical PC so you can use VDI technologies like Blast Extreme to remote in to physical desktops at your office. It makes the perfect remote access solution. Give it a try today with an evaluation license!

To get your evaluation license, please visit https://my.vmware.com/en/web/vmware/evalcenter?p=horizon-7.

Apr 042020
 
VMware Horizon View Icon

I see quite a bit of traffic come in on a regular basis pertaining to issues with VMware Horizon View. A lot of these visitors either are looking for help in setting something up or are experiencing an issue I’ve dealt with. While my posts usually help these people do specific things or troubleshoot specific issues, one of the biggest issues that comes up is when users experience a VMware Horizon blank screen (or black).

This can be caused by a number of different things. I wanted to take this opportunity to go over some of the most common issues that cause this and make a master guide for troubleshooting and fixing the VMware Horizon blank screen.

Horizon Blank Screen Causes

There’s a number of different causes of a blank or black screen when connecting and establishing a VDI session to Horizon View. Click on the item below to jump to that section of the post.

Now that we have a list, let’s dive in to each of these individually. Some of these will require you to do your own research and will only guide you, while other sections will include the full fix for the issue.

VMware Tools and Horizon Agent Installation Order

When deploying the VMware Horizon View agent, you are required to install the agent, along with VMware tools in a specific order. Failing to do so can cause problems, including a blank screen screen.

The installation order:

  1. Install GPU/vGPU drivers (if needed)
  2. Install VMware Tools Agent
  3. Install the VMware Horizon Agent
  4. Install the VMware User Environment Manager Agent (if needed)
  5. Install the VMware App Volumes Agent (if needed)

It is important to also consider this when upgrading the agents as well.

Network ports are blocked (Computer Firewall, Network Firewall)

For the VMware Horizon agent to function properly, ports must be accesible through your firewall, whether it’s the firewall on the VM guest, client computer, or network firewall.

The following ports are required for the VMware Horizon Agent when connecting directly to a View Connection Server.

DestinationNetwork ProtocolDestination PortDetails
Horizon Connection ServerTCP443Login, authentication, and connection to the VMware Connection Server.
Horizon AgentTCP22443Blast Extreme
UDP22443Blast Extreme
TCP4172PCoIP
UDP4172PCoIP
TCP3389RDP (Remote Desktop Protocol)
TCP9427Client Shared Drive redirection (CDR) and Multi-media redirection (MMR).
TCP32111USB Redirection (Optional), can be incorporated in to the Blast Extreme connection.
Network Ports Required for VMware Horizon View to View Connection Server

The following ports are required for the VMware Horizon Agent when connecting through a VMware Unified Access Gateway (UAG).

DestinationNetwork ProtocolDestination PortDetails
Unified Access GatewayTCP443Login, authentication, and connection to the Unified Access Gateway. This port/connection can also carry tunneled RDP, client drive redirection, and USB redirection traffic.
TCP4172PCoIP via PCoIP Secure Gateway
UDP4172PCoIP via PCoIP Secure Gateway
UDP443Optional for Login traffic. Blast Extreme will attempt a UDP login if there are issues establishing a TCP connection.
TCP8443Blast Extreme via Blast Secure Gateway (High Performance connection)
UDP8443Blast Extreme via Blast Secure Gateway (Adaptive performance connection)
TCP443Blast Extreme via UAG port sharing.
Network Ports Required for VMware Horizon View to VMware Unified Access Gateway (UAG)

You’ll notice the ports that are required for Blast Extreme and PCoIP. If these are not open you can experience a blank screen when connecting to the VMware Horizon VDI Guest VM.

For more information on VMware Horizon 7 network ports, visit https://techzone.vmware.com/resource/network-ports-vmware-horizon-7.

DNS Issues

While VMware Horizon View usually uses IP address for connectivity between the View Connection Server, guest VM, and client, I have seen times where DNS issues have stopped certain components from functioning properly.

It’s always a good idea to verify that DNS is functioning. DNS (forward and reverse) is required for VMware Horizon Linux guests VMs.

Incorrectly configured Unified Access Gateway

A big offender when it comes to blank screens is an incorrectly configured VMWare Unified Access Gateway.

Sometimes, first-time UAG users will incorrectly configure the View Connection server and UAG.

When configuring a UAG, you must disable both “Blast Secure Gateway”, and “PCoIP Secure Gateway” on the View Connection Server, as the UAG will be handling this. See below.

Picture of the Secure Gateway settings on VMware Horizon View Connection Server when used with VMware UAG.
Secure Gateway Settings on View Connection Server when used with UAG

Another regular issue is when admins misconfigure the UAG itself. There are a number of key things that must be configured properly. These are the values that should be populated on the UAG under Horizon Settings.

Connection Server URLhttps://ConnectionServerIP:443
Connection Server URL Thumbprintsha1=SSLTHUMPRINT
(Thumbprint of the SSL certificate your View Connection Server is using)
PCOIP External URLUAG-EXTERNALIP:4172
Blast External URLUAG-InternetFQDN:443
Tunnel External URLUAG-InternetFQDN:443

You must also have a valid SSL certificate installed under “TLS Server Certificate Settings”. I’d recommend applying it to both the admin and internal interface. This is a certificate that must match the FQDN (internal and external) of your UAG appliance.

Once you’re good, you’re green!

Picture of the VMware UAG interface showing all green (functioning).
VMware Unified Access Gateway showing valid

You should always see green lights, all protocols should work, and the connections should run smooth. If not, troubleshoot.

GPU Driver Issue

When using a GPU with your VM for 3D graphics, make sure you adhere to the requirements of the GPU vendor, along with the VMware requirements.

Some vendors have display count, resolution, and other limits that when reached, cause Blast Extreme to fail.

An incorrectly installed driver can also cause issues. Make sure that there are no issues with the drivers in the “Device Manager”.

VMware documents regarding 3D rendering:

Blast Extreme log files can be found on the guest VM in the following directory.

C:\ProgramData\VMware\VMware Blast\

Looking at these log files, you can find information that may pertain to the H.264 or display driver issues that will assist in troubleshooting.

VMware Tools

A corrupt VMware tools install, whether software or drivers can cause display issues. Make sure that the drivers (including the display driver) are installed and functioning properly.

It may be a good idea to completely uninstall VMware Tools and re-install.

If you’re experiencing display driver issues (such as a blank screen), before re-installing VMware Tools try forcibly removing the display driver.

  1. Open “Device Manager”
  2. Right click on the VMware Display adapter and open “Properties”
  3. On the “Driver” tab, select “Uninstall”
  4. Check the box for “Delete the driver software for this device”.

This will fully remove the VMware driver. Now re-install VMware Tools.

Horizon Agent

Often, re-installing the Horizon Agent can resolve issues. Always make sure that VMware Tools are installed first before installing the Horizon Agent.

Make sure that if you are running 64-bit Windows in the VM then you install and use the 64-bit Horizon Agent.

You may experience issues with the “VMware Horizon Indirect Display Driver”. Some users have reported an error on this driver and issues loading it, resulting in a blank screen. To do this, I’d recommend forcibly uninstalling the driver and re-installing the Horizon Agent.

To forcibly remove the “VMware Horizon Indirect Display Driver”:

  1. Open “Device Manager”
  2. Right click on the “VMware Horizon Indirect Display Driver” and open “Properties”
  3. On the “Driver” tab, select “Uninstall”
  4. Check the box for “Delete the driver software for this device”.

Now proceed to uninstall and reinstall the Horizon View Agent.

On a final note, when running the Horizon Agent on Horizon for Linux, make sure that forward and reverse DNS entries exist, and that DNS is functioning on the network where the Linux VM resides.

Video Settings (Video Memory (VRAM), Resolution, Number of Displays)

When experiencing video display issues or blank screens on VMware Horizon View, these could be associated with the guest VM’s memory, video memory (VRAM), display resolution, and number of displays.

Make sure you are adhering to the specifications put forth by VMware. Please see the following links for more information.

Protocol

When troubleshooting blank screens with VMware Horizon, you need to try to identify if it’s specific to the guest VM, or if it’s associated with the connection protocol you’re using (and the route it takes whether through a Connection Server, or UAG).

Always try different protocols to see if the issue is associated with all, or one. Then try establishing connections and find if it’s isolated direct to the Connection Server, or through the UAG.

If the issue is with a specific protocol, you can view the protocol log files. If the issue is with the UAG, you can troubleshoot the UAG.

Log files can be found in the following directory:

C:\ProgramData\VMware\

HTTPS Proxy and redirection issues

If you are connecting through a network that does passive HTTPS scanning or that uses a proxy server, you may experience issues with inability to connect, or blank screens.

You’ll need to modify your firewall or proxy to allow the VMware connection and open the required ports for VMware Horizon View.

Login banner or disclaimer (PCoIP)

I haven’t seen or heard of this one in some time, but when using VMware Horizon with PCoIP, sessions can fail or show a blank screen when the legal disclaimer login banner is used.

For more information on this issue, and how to resolve or workaround, visit https://kb.vmware.com/s/article/1016961.

Old version of Horizon View

It never stops surprising me how old some of the VMware Horizon View environments are that some businesses are running. VMware regularly updates, and releases new versions of VMware Horizon View that resolve known issues and bugs in the software.

While it may be difficult, simply upgrading your VMware Horizon environment (VMware vSphere, View Connection Server, VMware Tools, VMware Horizon Agent) can resolve your issues.

Blank Screen connecting to Physical PC running Horizon Agent

When you install the VMware Horizon Agent on a Physical PC, you may encounter issues with a blank screen.

This is usually caused by:

After troubleshooting these issues, you should be able to resolve the issue.

Conclusion

As you can see there are a number of different things that can cause Horizon View to show a blank screen on login.

Let me know if this helped you out, or if you find other reasons and feel I should add them to the list!