Oct 152020
 
VMware vCLS VM in VM List

Did a new VM appear on your VMware vSphere cluster called “vCLS”? Maybe multiple appeared named “vCLS (1)”, “vCLS (2)”, “vCLS (3)” appeared.

VMware vCLS VM in vSphere Cluster Objects
VMware vCLS VM in vSphere Cluster Objects

This could be frightening but fear not, this is part of VMware vSphere 7.0 Update 1.

What is the vCLS VM?

The vCLS virtural machine is essentially an “appliance” or “service” VM that allows a vSphere cluster to remain functioning in the event that the vCenter Server becomes unavailable. It will maintain the health and services of that cluster.

Where did the vCLS VM come from?

The vCLS VM will appear after upgrading to vSphere 7.0 Update 1. I’m assuming it was deployed during the upgrade process.

It does not appear in the standard Cluster, Hosts, and VMs view, but does appear when looking at the vSphere objects VM lists, Storage VM lists, etc…

Is it normal to have more than one vCLS VM?

The vCLS VMs are created when hosts are added to a vSphere Cluster. Up to 3 vCLS VMs are required to run in each vSphere Cluster.

The vCLS VMs will also appear on clusters which contain only one or two hosts. These configurations will result in either 1 or 2 vCLS VMs named “vCLS (1)” and “vCLS (2)”.

A note on licensing in regards to the vCLS VM

For VMware environments that use VM based licensing like vSphere for ROBO (Remote Office Branch Office), vCLS VMs are shown in the licensing interface as counting towards licensed VMs. Please Note that these VMs do not official count towards your purchased licenses as these are VMware System VMs. Please read VMware KB 80472 for more information on this.

More Information on vCLS VMs

For more information and technical specifics, you can visit the link below:

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenterhost.doc/GUID-96BD6016-4BE7-4B1C-8269-568D1555B08C.html

Hope this posts helps, and puts some minds at ease. Your VMware environment has NOT been compromised.

Oct 102020
 

If you’re like me and use an older Nvidia GRID K1 or K2 vGPU video card for your VDI homelab, you may notice that when using VMware Horizon that VMware Blast h264 encoding is no longer being offloaded to the GPU and is instead being encoded via the CPU.

The Problem

Originally when an environment was configured with an Nvidia GRID K1 or K2 card, not only does the card provide 3D acceleration and rendering, but it also offloads the VMware BLAST h264 stream (the visual session) so that the CPU doesn’t have to. This results in less CPU usage and provides a streamlined experience for the user.

This functionality was handled via NVFBC (Nvidia Frame Buffer Capture) which was part of the Nvidia Capture SDK (formerly known as GRID SDK). This function allowed the video card to capture the video frame buffer and encode it using NVENC (Nvidia Encoder).

Ultimately after spending hours troubleshooting, I learned that NVFBC has been deprecated and is no longer support, hence why it’s no longer functioning. I also checked and noticed that tools (such as nvfbcenable) were no longer bundled with the VMware Horizon agent. One can assume that the agent doesn’t even attempt to check or use this function.

Symptoms

Before I was aware of this, I noticed that while 3D Acceleration and graphics were functioning, I was experiencing high CPU usage. Upon further investigation I noticed that my VMware BLAST sessions were not offloading h264 encoding to the video card.

VMware Horizon Performance Tracker
VMware Horizon Performance Tracker with NVidia GRID K1

You’ll notice above that under the “Encoder” section, the “Encoder Name” was listed as “h264 4:2:0”. Normally this would say “NVIDIA NvEnc H264” (or “NVIDIA NvEnc HEVC” on newer cards) if it was being offloaded to the GPU.

Looking at a VMware Blast session (Blast-Worker-SessionId1.log), the following lines can be seen.

[INFO ] 0x1f34 bora::Log: NvEnc: VNCEncodeRegionNvEncLoadLibrary: Loaded NVIDIA SDK shared library "nvEncodeAPI64.dll"
[INFO ] 0x1f34 bora::Log: NvEnc: VNCEncodeRegionNvEncLoadLibrary: Loaded NVIDIA SDK shared library "nvml.dll"
[WARN ] 0x1f34 bora::Warning: GetProcAddress: Failed to resolve nvmlDeviceGetEncoderCapacity: 127
[WARN ] 0x1f34 bora::Warning: GetProcAddress: Failed to resolve nvmlDeviceGetProcessUtilization: 127
[WARN ] 0x1f34 bora::Warning: GetProcAddress: Failed to resolve nvmlDeviceGetGridLicensableFeatures: 127
[INFO ] 0x1f34 bora::Log: NvEnc: VNCEncodeRegionNvEncLoadLibrary: Some NVIDIA nvml functions unavailable, unloading
[INFO ] 0x1f34 bora::Log: NvEnc: VNCEncodeRegionNvEncUnloadLibrary: Unloading NVIDIA SDK shared library "nvEncodeAPI64.dll"
[INFO ] 0x1f34 bora::Log: NvEnc: VNCEncodeRegionNvEncUnloadLibrary: Unloading NVIDIA SDK shared library "nvml.dll"
[WARN ] 0x1f34 bora::Warning: GetProcAddress: Failed to resolve nvmlDeviceGetEncoderCapacity: 127
[WARN ] 0x1f34 bora::Warning: GetProcAddress: Failed to resolve nvmlDeviceGetProcessUtilization: 127
[WARN ] 0x1f34 bora::Warning: GetProcAddress: Failed to resolve nvmlDeviceGetGridLicensableFeatures: 127

You’ll notice it tries to load the proper functions, however it fails.

The Solution

Unfortunately the only solution is to upgrade to newer or different hardware.

The GRID K1 and GRID K2 cards have reached their EOL (End of Life) and are no longer support. The drivers are not being maintained or updated so I doubt they will take advantage of the newer frame buffer capture functions of Windows 10.

Newer hardware and solutions have incorporated this change and use a different means of frame buffer capture.

To resolve this in my own homelab, I plan to migrate to an AMD FirePro S7150x2.

Oct 062020
 
Video Capture of The Tech Informative

A joint project of mine with Rob Dalton, The Tech Informative is a video podcast on YouTube dedicated to everything tech with a focus on corporate IT.

We intend to keep it informal, fun, and packed full of handy and useful information.

Check it out, and don’t forget to like and subscribe. We also appreciate suggestions and feedback!

This introduction episode is the first of many to come!

Jul 222020
 
VMware Logon

When upgrading from any version of VMware vCSA to version 7.0, you may encounter a problem during the migration phase and be asked to specifiy a new “Export Directory”.

I’ve seen this occur on numerous upgrades and often find the same culprit causing the issue. I’ve found a very simple fix compared to other solutions online.

The full prompt for this issue is: “Enter a new export directory on the source machine below”

The Problem

When you upgrade the vCenter vCSA, the process migrates all data over from the source appliance, to the new vCSA 7 appliance.

This data can include the following (depending on your selection):

  • Configuration
  • Configuration and historical data (events and tasks)
  • Configuration and historical data (events, tasks, and performance metric)
  • vSphere Update Manager (updates, configuration, etc.)

This data can accumulate, especially the VMware vSphere Update Manager.

In the most recent upgrade I performed, I noticed that the smallest option (configuration only) was around 8GB, which is way over the 4.7GB default limit.

Could it be vSphere Update Manager?

I’ve seen VMware VUM cause numerous issues over the years with upgrades. VUM has caused issues upgrading from earlier versions to 6.x, and in this case it caused this issue upgrading to vCSA 7.x as well.

In my diagnosis, I logged in to the SSH console of the source appliance, and noticed that the partitions containing the VUM data (which includes update files) was around 7.4GB. This is the “/storage/updatemgr/” partition.

I wasn’t sure if this was included, but the 8GB of configuration, minus the 7.4GB of VUM data, could technically get me to around 0.6GB for migration if this was in fact included.

In my environment, I have the default (and simple) implementation of VUM with the only customization being the HPE VIBs depot. I figured maybe I should blast away the VUM and start from scratch on VMware vCSA 7.0 to see if this fixes the issue.

The Fix

To fix this issue, I simply completely reset the VMware Update Manager Database.

For details on this process and before performing these steps, please see VMware KB 2147284.

Let’s get to it:

  1. Close the migration window (you can reopen this later)
  2. Log in to your vCSA source appliance via SSH or console
  3. Run the applicable steps as defined in the VMware KB 2147284 to reset VUM (WARNING: commands are version specific). In my case on vCSA 6.5 I ran the following commands:
    1. shell
    2. service-control --stop vmware-updatemgr
    3. /usr/lib/vmware-updatemgr/bin/updatemgr-util reset-db
    4. rm -rf /storage/updatemgr/patch-store/*
    5. service-control --start vmware-updatemgr
  4. Open your web broswer and navigate to https://new-vcsa-IP:5480 and resume the migration. You will now notice a significant space reduction and won’t need to specify a new mount point.

That’s it! You have a shiny new clean VUM instance, and can successfully upgrade to vCSA 7.0 without having to specify a new mount point.

To reconfigure and restore any old configuration to VUM, you’ll do so in the “VMware Lifecycle Management” section of the VMware vCenter Server Appliance interface.

Alternatively, in the rare event it’s not related to the VUM data, you can set the export directory to somewhere in “/tmp/” which is another workaround this issue which may allow you to continue.

Jul 132020
 
Picture of the DUO Security Logo

When you’re looking for additional or enhanced options to secure you’re business and enterprise IT systems, MFA/2FA can help you achieve this. Get away from the traditional single password, and implement additional means of authentication! MFA provides a great compliment to your cyber-security policies.

My company, Digitally Accurate Inc, has been using the Duo Security‘s MFA product in our own infrastructure, as well as our customers environments for some time. Digitally Accurate is a DUO Partner and can provide DUO MFA Services including licensing/software and the hardware tokens (Duo D-100 Tokens using HOTP).

What is MFA/2FA

MFA is short for Multi Factor authentication, additionally 2FA is short for Two Factor Authentication. While they are somewhat the same, multi means many, and 2 means two. Additional security is provided with both, since it provides more means of authentication.

Traditionally, users authenticate with 1 (one) level of authentication: their password. In simple terms MFA/2FA in addition to a password, provides a 2nd method of authentication and identity validation. By requiring users to authentication with a 2nd mechanism, this provides enhanced security.

Why use MFA/2FA

In a large portion of security breaches, we see users passwords become compromised. This can happen during a phishing attack, virus, keylogger, or other ways. Once a malicious user or bot has a users credentials (username and password), they can access resources available to that user.

By implementing a 2nd level of authentication, even if a users password becomes compromised, the real (or malicious user) must pass a 2nd authentication check. While this is easy for the real user, in most cases it’s nearly impossible for a malicious user. If a password get’s compromised, nothing can be accessed as it requires a 2nd level of authentication. If this 2nd method is a cell phone or hardware token, a malicious user won’t be ale to access the users resources unless they steal the cell phone, or hardware token.

How does MFA/2FA work

When deploying MFA or 2FA you have the option of using an app, hardware token (fob), or phone verification to perform the additional authentication check.

After a user attempts to logs on to a computer or service with their username and password, the 2nd level of authentication will be presented, and must pass in order for the login request to succeed.

Please see below for an example of 2FA selection screen after a successful username and password:

Screenshot of Duo MFA 2FA Prompt on Windows Login
Duo Security Windows Login MFA 2FA Prompt

After selecting an authentication method for MFA or 2FA, you can use the following

2FA with App (Duo Push)

Duo Push sends an authentication challenge to your mobile device which a user can then approve or deny.

Please see below for an example of Duo Push:

Screenshot of Duo Push Notification to Mobile Android App
Duo Push to Mobile App on Android

Once the user selects to approve or deny the login request, the original login will either be approved or denied. We often see this as being the preferred MFA/2FA method.

2FA with phone verification (Call Me)

Duo phone verification (Call Me) will call you on your phone number (pre-configured by your IT staff) and challenge you to either hangup to deny the login request, or press a button on the keypad to accept the login request.

While we rarely use this option, it is handy to have as a backup method.

2FA with Hardware Token (Passcode)

Duo Passcode challenges are handled using a hardware token (or you can generate a passcode using the Duo App). Once you select this method, you will be prompted to enter the passcode to complete the 2FA authentication challenge. If you enter the correct passcode, the login will be accepted.

Here is a Duo D-100 Token that uses HOTP (HMAC-based One Time Password):

Picture of Duo D-100 HOTP Hardware Token
Duo D-100 HOTP Hardware Token

When you press the green button, a passcode will be temporarily displayed on the LCD display which you can use to complete the passcode challenge.

You can purchase Hardware Token’s directly from Digitally Accurate Inc by contacting us, your existing Duo Partner, or from Duo directly. Duo is also compatible with other 3rd party hardware tokens that use HOTP and TOTP.

2FA with U2F

While you can’t visibly see the option for U2F, you can use U2F as an MFA or 2FA authentication challenge. This includes devices like a Yubikey from Yubico, which plugs in to the USB port of your computer. You can attach a Yubikey to your key chain, and bring it around with you. The Yubikey simply plugs in to your USB port and has a button that you press when you want to authenticate.

When the 2FA window pops up, simply hit the button and your Yubikey will complete the MFA/2FA challange.

What can MFA/2FA protect

Duo MFA supports numerous cloud and on-premise applications, services, protocols, and technologies. While the list is very large (full list available at https://duo.com/product/every-application), we regularly deploy and use Duo Security for the following configurations.

Windows Logins (Server and Workstation Logon)

Duo MFA can be deployed to not only protect your Windows Servers and Workstations, but also your remote access system as well.

When logging on to a Windows Server or Windows Workstation, a user will be presented with the following screen for 2FA authentication:

Screenshot of Duo MFA 2FA Prompt on Windows Login
Duo Security Windows Login MFA 2FA Prompt

Below you can see a video demonstration of DUO on Windows Login.

DUO works with both Windows Logins and RDP (Remote Desktop Protocol) Logins.

VMWare Horizon View Clients (VMWare VDI Logon)

Duo MFA can be deployed to protect your VDI (Virtual Desktop Infrastructure) by requiring MFA or 2FA when users log in to access their desktops.

When logging on to the VMware Horizon Client, a user will be presented with the following screen for 2FA authentication:

Screenshot of Duo MFA 2FA Prompt on VMWare Horizon Client Login
Duo Security VMWare Horizon Client Login MFA 2FA Prompt

Below you can see a video demonstration of DUO on VMware Horizon View (VDI) Login.

Sophos UTM (Admin and User Portal Logon)

Duo MFA can be deployed to protect your Sophos UTM firewall. You can protect the admin account, as well as user accounts when accessing the user portal.

If you’re using the VPN functionality on the Sophos UTM, you can also protect VPN logins with Duo MFA.

Unix and Linux (Server and Workstation Logon)

Duo MFA can be deployed to protect your Unix and Linux Servers. You can protect all user accounts, including the root user.

We regularly deploy this with Fedora and CentOS (even FreePBX) and you can protect both SSH and/or console logins.

When logging on to a Unix or Linux server, a user will be presented with the following screen for 2FA authentication:

Screenshot of Duo MFA 2FA Prompt on CentOS Linux Login
Duo Security CentOS Linux login MFA 2FA Prompt

Below you can see a video demonstration of DUO on Linux.

WordPress Logon

Duo MFA can be deployed to protect your WordPress blog. You can protect your admin and other user accounts.

If you have a popular blog, you know how often bots are attempting to hack and brute force your passwords. If by chance your admin password becomes compromised, using MFA or 2FA can protect your site.

When logging on to a WordPress blog admin interface, a user will be presented with the following screen for 2FA authentication:

Screenshot of Duo MFA 2FA Prompt on WordPress Login
Duo Security WordPress Login MFA 2FA Prompt

Below you can see a video demonstration of DUO on a WordPress blog.

How easy is it to implement

Implementing Duo MFA is very easy and works with your existing IT Infrastructure. It can easily be setup, configured, and maintained on your existing servers, workstations, and network devices.

Duo offers numerous plugins (for windows), as well as options for RADIUS type authentication mechanisms, and other types of authentication.

How easy is it to manage

Duo is managed through the Duo Security web portal. Your IT admins can manage users, MFA devices, tokens, and secured applications via the web interface. You can also deploy appliances that allow users to manage, provision, and add their MFA devices and settings.

Duo also integrates with Active Directory to make managing and maintaining users easy and fairly automated.

Let’s get started with Duo MFA

Want to protect your business with MFA? Give me a call today!