May 052019
 
Ubuntu Orange Logo

After upgrading a computer from Ubuntu 16.04 LTS to Ubuntu 18.04 LTS, during boot the screen goes blank (turns black), all HD disk activity halts, and the system becomes frozen. This event can also occur on a fresh installation or when updates are installed.

This is due to a video mode issue that causes the system to halt or freeze. It’s much like the issue I described here on a Fedora Linux system.

Temporary Fix

To get the system to boot:

  1. After turning on your PC, hold the right SHIFT key to get to the GRUB bootloader.
  2. Once GRUB is open, press the “e” key to edit the first highlighted entry “Ubuntu”.
  3. Move your cursor down to the line that starts with “linux”, and use the right arrow key to find the section with the words “ro quiet splash”.
  4. Add “nomodeset” after these words.
    nomodeset
  5. Feel free to remove “quiet” and “splash” for more verbosity to troubleshoot the boot process.
  6. Press “CTRL + X” or “F10” to boot.
  7. The system should now boot.

Permanent Fix

To permanently resolve the issue:

  1. Once the system has booted using the temporary fix, log in.
  2. Open a terminal window (Applications -> Terminal, or press the “Start” button and type terminal).
  3. Either “su” in to root, or use “sudo” to open your favorite text editor and edit the file “/etc/default/grub” (I use nano which can be install by running “dnf install nano”):
    nano /etc/default/grub
  4. Locate the line with the variable “GRUB_CMDLINE_LINUX_DEFAULT”, and add “nomodeset” to the variables. Feel free to remove “splash” and “quiet” if you’d like text boot. Here’s an example of my line after editing (yours will look different):
    GRUB_CMDLINE_LINUX_DEFAULT="quiet splash nomodeset"
  5. Save the file and exit the text editor (CTRL+X to quit, the press “y” and enter to save).
  6. At the bash prompt, execute the following command to regenerate the grub.conf file on the /boot partition from your new default file:
    update-grub
  7. Restart your system, it should now boot!

Please Note: Always make sure you have a full system backup before modifying any system files!

May 042019
 
Ubuntu Orange Logo

You’re trying to install Ubuntu on your computer, but it freezes due to lack of resources, specifically memory. This can happen when you’re trying to re-purpose old laptops, netbooks, etc.

This recently happened to met as I tried to install Ubuntu on an old HP Netbook. Originally I used Fedora, but had to switch to Ubuntu due to library issues (I wanted to use the VMware Horizon Client on it).

Unfortunately, when I’d kick off the USB installer, the OS would completely freeze (mouse either unresponsive, or extremely glitchy).

The Fix – External SWAP File

In the ~5 minutes where the system is operable, I used the key sequence “CTRL + ALT + F2” to get to a text tty console session. From here I noticed the system eventually uses all the RAM and maxes out the memory. When this occurs, this is when the system becomes unresponsive.

Since this is a Live CD installer, there is no swap file for the system to use once the RAM has filled up.

To fix this and workaround the problem, I grabbed a second blank USB stick and used it as an external swap file. Using this allowed me to run the installer, complete the installer, and successfully install Ubuntu.

Please make sure you are choosing the right device names in the instructions below. Choosing the wrong device name can cause your to write to the wrong USB stick, or worse the hard drive of your system.

Instructions:

  1. Attached USB Installer, boot system.
  2. Once system has booted, press “CTRL + ALT + F2” to open a tty console session.
  3. Login using user: “Ubuntu” with a blank password.
  4. Type “sudo su” to get a root shell.
  5. Type in “tail -f /var/log/kern.log” and connect your spare blank USB stick that you want to use for SWAP space. Note the device name, in my case it was “/dev/sdd”.
  6. Press “CTRL + C” to stop tailing the log file, then run “fdisk /dev/sdd” and replace “/dev/sdd” with whatever your device was. PLEASE MAKE SURE YOU ARE CHOOSING THE RIGHT USB DEVICE NAME.
  7. Use “n” to create a new partition, follow the prompts, when it asks for size I randomly chose “+2G” for a 2GB swap file. Use “w” to write the partition table and then quit the fdisk application.
  8. Run “mkswap /dev/sdd1” and replace “sdd1” with the device and partition number of your USB Swap stick. This will format the partition and mark it as a SWAP filesystem.
  9. Run “swapon /dev/sdd1” and replace “sdd1” with your swap partition you created. This will activate the external swap file on the USB stick.
  10. Press “CTRL + ALT + F1” to return to the Ubuntu installation guide. Continue the install as normal.

This should also work for other Linux distributions, as I have also used this in the past with Fedora (on a Single Board Computer with almost no RAM).

During the install process where the Ubuntu installer formats your hard drive, the install will actually mount the hard drive swap file as well (it’ll use both). Once the installer is complete, shut down the system and remove the USB SWAP stick.

May 032019
 
Ubuntu Enterprise Logo

So here we’re going to talk about Installing the VMware Horizon Agent for Linux on an Ubuntu 18.04 Guest VM for use with VDI. Ultimately you’ll be setting up Horizon 7 for Linux Desktops.

This will allow you to add Linux VMs to your VDI environment.

Ubuntu 18.04 LTS running on VMware Hoirzon Client using Horizon for Linux
Horizon for Linux on Ubuntu 18.04 LTS

VMware has the documentation here, but I’ve condensed this down for your to get up and running quickly, as well as deal with a few bugs I noticed.

Requirements

You’ll need the following to get started

  • VMware Horizon View 7 (I’m, using 7.8)
  • Horizon Enterprise or Horizon for Linux Valid Licensing
  • Horizon VDI environment that’s functioning and working
  • Ubuntu 18.04 LTS Installer ISO (download here)
  • Horizon Agent for Linux (download here)
  • Functioning internal DNS

Once you have the above, we can get going!

Instructions

  1. Create a static DHCP reservation, or note the IP you will set statically to the Ubuntu VM
  2. Create a host entry on your internal DNS Server so that the IP of your Ubuntu VM will have a functioning internal FQDN
  3. Create a VM and Install Ubuntu 18.04 TLS using the ISO you downloaded above
  4. If you’re using a static IP, set this now. If you’re using a DHCP reservation, make sure it’s working
  5. Install any Root CA’s or modifications you need for network access (usually not needed unless you’re on an enterprise network)
  6. Open a terminal, and type “sudo su” to get a root console
  7. Run the following command
    apt install open-vm-tools-desktop openssh-server python python-dbus python-gobject lightdm
    You can skip the “openssh-server” package if you don’t want to enable SSH. A display manager configuration prompt will present itself, choose “gdm”
  8. Now we need to add the internal FQDN to the hosts file. Run “nano /etc/hosts” to open the hosts file. Create a new line at the top and enter
    127.0.0.1 compname.domain.com compname
    Modify “compname.domain.com” and “compname” to reflect your FQDN and computer name.
  9. Restart the Guest VM
  10. Open terminal, “sudo su” to get a root console
  11. Extract the Horizon Agent tarball with
    tar zxvf VMware-horizonagent-linux-x86_64-7.8.0-12610615.tar.gz
    Please note that if your version is different, your file name may be different. Please adjust accordingly.
  12. Change directory in to the VMware Horizon Agent that we just extracted.
  13. Run the installer for the horizon agent with
    ./install_viewagent.sh
  14. Follow the prompts, restart the host
  15. Log on to your View Connection Server
  16. Create a manual pool, and configure it accordingly
  17. Add the Ubuntu Linux VM to the pool
  18. Entitle the users to the pool, and assign the users to the host under inventory

Final Notes

In the VMware documentation, it states to select “lightdm” on the Display MAnager configuration window that presents itself in step 7. However if you choose this, the VMware Horizon Agent for Linux will not install. Choosing “gdm” allows it to install and function.

I have noticed audio issues when using the Spotify snapd. I believe this is caused by timer-based audio scheduling in PulseAudio. I have tried using the “tsched=0” flag in the PulseAudio config, however this has no effect and I haven’t been able to resolve this yet. Audio in Chrome and other audio players works fine. A workaround is to install “pavucontrol” and have it open while using Spotify and the audio issues will temporarily be resolved. I also tried using the VMware Tools (deprecated) instead of OpenVM Tools to see if this helped with the audio issues, but it did not.

If you have 3D Acceleration with a GRID card, the Linux VDI VM will be able to utilize 3D accelerated vSGA as long as you have it configured on the ESXi host.

May 022019
 
Nvidia GRID Logo

I can’t tell you how excited I am that after many years, I’ve finally gotten my hands on and purchased an Nvidia Quadro K1 GPU. This card will be used in my homelab to learn, and demo Nvidia GRID accelerated graphics on VMware Horizon View. In this post I’ll outline the details, installation, configuration, and thoughts. And of course I’ll have plenty of pictures below!

The focus will be to use this card both with vGPU, as well as 3D accelerated vSGA inside in an HPe server running ESXi 6.5 and VMware Horizon View 7.8.

Please Note: Some, most, or all of what I’m doing is not officially supported by Nvidia, HPe, and/or VMware. I am simply doing this to learn and demo, and there was a real possibility that it may not have worked since I’m not following the vendor HCL (Hardware Compatibility lists). If you attempt to do this, or something similar, you do so at your own risk.

Nvidia GRID K1 Image

For some time I’ve been trying to source either an Nvidia GRID K1/K2 or an AMD FirePro S7150 to get started with a simple homelab/demo environment. One of the reasons for the time it took was I didn’t want to spend too much on it, especially with the chances it may not even work.

Essentially, I have 3 Servers:

  1. HPe DL360p Gen8 (Dual Proc, 128GB RAM)
  2. HPe DL360p Gen8 (Dual Proc, 128GB RAM)
  3. HPe ML310e Gen8 v2 (Single Proc, 32GB RAM)

For the DL360p servers, while the servers are beefy enough, have enough power (dual redundant power supplies), and resources, unfortunately the PCIe slots are half-height. In order for me to use a dual-height card, I’d need to rig something up to have an eGPU (external GPU) outside of the server.

As for the ML310e, it’s an entry level tower server. While it does support dual-height (dual slot) PCIe cards, it only has a single 350W power supply, misses some fancy server technologies (I’ve had issues with VT-d, etc), and only a single processor. I should be able to install the card, however I’m worried about powering it (it has no 6pin PCIe power connector), and having ESXi be able to use it.

Finally, I was worried about cooling. The GRID K1 and GRID K2 are typically passively cooled and meant to be installed in to rack servers with fans running at jet engine speeds. If I used the DL360p with an external setup, this would cause issues. If I used the ML310e internally, I had significant doubts that cooling would be enough. The ML310e did have the plastic air baffles, but only had one fan for the expansion cards area, and of course not all the air would pass through the GRID K1 card.

The Purchase

Because of a limited budget, and the possibility I may not even be able to get it working, I didn’t want to spend too much. I found an eBay user local in my city who had a couple Grid K1 and Grid K2 cards, as well as a bunch of other cool stuff.

We spoke and he decided to give me a wicked deal on the Grid K1 card. I thought this was a fantastic idea as the power requirements were significantly less (more likely to work on the ML310e) on the K1 card at 130 W max power, versus the K2 card at 225 W max power.

NVIDIA GRID K1 and K2 Specifications
NVIDIA GRID K1 and K2 Specification Table

The above chart is a capture from:
https://www.nvidia.com/content/cloud-computing/pdf/nvidia-grid-datasheet-k1-k2.pdf

We set a time and a place to meet. Preemptively I ran out to a local supply store to purchase an LP4 power adapter splitter, as well as a LP4 to 6pin PCIe power adapter. There were no available power connectors inside of the ML310e server so this was needed. I still thought the chances of this working were slim…

These are the adapters I purchased:

Preparation and Software Installation

I also decided to go ahead and download the Nvidia GRID Software Package. This includes the release notes, user guide, ESXi vib driver (includes vSGA, vGPU), as well as guest drivers for vGPU and pass through. The package also includes the GRID vGPU Manager. The driver I used was from:
https://www.nvidia.com/Download/driverResults.aspx/144909/en-us

To install, I copied over the vib file “NVIDIA-vGPU-kepler-VMware_ESXi_6.5_Host_Driver_367.130-1OEM.650.0.0.4598673.vib” to a datastore, enabled SSH, and then ran the following command to install:

esxcli software vib install -v /path/to/file/NVIDIA-vGPU-kepler-VMware_ESXi_6.5_Host_Driver_367.130-1OEM.650.0.0.4598673.vib

The command completed successfully and I shut down the host. Now I waited to meet.

We finally met and the transaction went smooth in a parking lot (people were staring at us as I handed him cash, and he handed me a big brick of something folded inside of grey static wrap). The card looked like it was in beautiful shape, and we had a good but brief chat. I’ll definitely be purchasing some more hardware from him.

Hardware Installation

Installing the card in the ML310e was difficult and took some time with care. First I had to remove the plastic air baffle. Then I had issues getting it inside of the case as the back bracket was 1cm too long to be able to put the card in. I had to finesse and slide in on and angle but finally got it installed. The back bracket (front side of case) on the other side slid in to the blue plastic case bracket. This was nice as the ML310e was designed for extremely long PCIe expansion cards and has a bracket on the front side of the case to help support and hold the card up as well.

For power I disconnected the DVD-ROM (who uses those anyways, right?), and connected the LP5 splitter and the LP5 to 6pin power adapter. I finally hooked it up to the card.

I laid the cables out nicely and then re-installed the air baffle. Everything was snug and tight.

Please see below for pictures of the Nvidia GRID K1 installed in the ML310e Gen8 V2.

Host Configuration

Powering on the server was a tense moment for me. A few things could have happened:

  1. Server won’t power on
  2. Server would power on but hang & report health alert
  3. Nvidia GRID card could overheat
  4. Nvidia GRID card could overheat and become damaged
  5. Nvidia GRID card could overheat and catch fire
  6. Server would boot but not recognize the card
  7. Server would boot, recognize the card, but not work
  8. Server would boot, recognize the card, and work

With great suspense, the server powered on as per normal. No errors or health alerts were presented.

I logged in to iLo on the server, and watched the server perform a BIOS POST, and start it’s boot to ESXi. Everything was looking well and normal.

After ESXi booted, and the server came online in vCenter. I went to the server and confirmed the GRID K1 was detected. I went ahead and configured 2 GPUs for vGPU, and 2 GPUs for 3D vSGA.

ESXi Graphics Settings for Host Graphics and Graphics Devices
ESXi Host Graphics Devices Settings

VM Configuration

I restarted the X.org service (required when changing the options above), and proceeded to add a vGPU to a virtual machine I already had configured and was using for VDI. You do this by adding a “Shared PCI Device”, selecting “NVIDIA GRID vGPU”, and I chose to use the highest profile available on the K1 card called “grid_k180q”.

Virtual Machine Edit Settings with NVIDIA GRID vGPU and grid_k180q profile selected
VM Settings to add NVIDIA GRID vGPU

After adding and selecting ok, you should see a warning telling you that must allocate and reserve all resources for the virtual machine, click “ok” and continue.

Power On and Testing

I went ahead and powered on the VM. I used the vSphere VM console to install the Nvidia GRID driver package (included in the driver ZIP file downloaded earlier) on the guest. I then restarted the guest.

After restarting, I logged in via Horizon, and could instantly tell it was working. Next step was to disable the VMware vSGA Display Adapter in the “Device Manager” and restart the host again.

Upon restarting again, to see if I had full 3D acceleration, I opened DirectX diagnostics by clicking on “Start” -> “Run” -> “dxdiag”.

DirectX Diagnostic Tool (dxdiag) showing Nvidia Grid K1 on VMware Horizon using vGPU k180q profile
dxdiag on GRID K1 using k180q profile

It worked! Now it was time to check the temperature of the card to make sure nothing was overheating. I enabled SSH on the ESXi host, logged in, and ran the “nvidia-smi” command.

nvidia-smi command on ESXi host showing GRID K1 information, vGPU information, temperatures, and power usage
“nvidia-smi” command on ESXi Host

According to this, the different GPUs ranged from 33C to 50C which was PERFECT! Further testing under stress, and I haven’t gotten a core to go above 56. The ML310e still has an option in the BIOS to increase fan speed, which I may test in the future if the temps get higher.

With “nvidia-smi” you can see the 4 GPUs, power usage, temperatures, memory usage, GPU utilization, and processes. This is the main GPU manager for the card. There are some other flags you can use for relevant information.

nvidia-smi with vgpu flag for vgpu information
“nvidia-smi vgpu” for vGPU Information
nvidia-smi with vgpu -q flag
“nvidia-smi vgpu -q” to Query more vGPU Information

Final Thoughts

Overall I’m very impressed, and it’s working great. While I haven’t tested any games, it’s working perfect for videos, music, YouTube, and multi-monitor support on my 10ZiG 5948qv. I’m using 2 displays with both running at 1920×1080 for resolution.

I’m looking forward to doing some tests with this VM while continuing to use vGPU. I will also be doing some testing utilizing 3D Accelerated vSGA.

The two coolest parts of this project are:

  • 3D Acceleration and Hardware h.264 Encoding on VMware Horizon
  • Getting a GRID K1 working on an HPe ML310e Gen8 v2

Leave a comment and let me know what you think! Or leave a question!

May 012019
 
VMware Horizon View Logo

One really cool feature that was released in VMware Horizon View 7.7, was the ability to install the Horizon Agent on to a Physical PC or Physical Workstation and use the Blast Extreme protocol. It even supports 3D Acceleration via a GPU!

As a system admin, I see value in having some Physical PCs managed by the View connection server.

I’ll be detailing some information about doing this, what’s required, what works, and what doesn’t below…

From the “What’s new in Horizon 7.7” doc at https://blogs.vmware.com/euc/2018/12/whats-new-horizon-7-7.html

You can now use the Blast Extreme display protocol to access physical PCs and workstations. Some limitations apply.

Additional information from the “Horizon 7.7 Release Notes” at https://docs.vmware.com/en/VMware-Horizon-7/7.7/rn/horizon-77-view-release-notes.html

Physical PCs and workstations with Windows 10 1803 Enterprise or higher can be brokered through Horizon 7 via Blast Extreme protocol.

Requirements

So here’s what’s required to get going:

  • Windows 10 Enterprise (Enterprise license is a must)
  • Physical PC or Workstation
  • VMware Horizon Licensing
  • VMware Horizon 7.7 Connection Server
  • VMware Horizon 7.7 Agent on Physical PC/Workstation
  • Manual Desktop Pool (Manual is required for Physical PCs to be added)

What Works

  • Blast Extreme
  • 3D Acceleration (via GPU with drivers)
  • 3D Acceleration with Consumer GPUs
  • Multiple Displays
  • Multiple GPUs

What Doesn’t Work

  • GPU Hardware h.264 encoding on consumer GPUs (h.264 encoding is still handled by the CPU)

Thoughts

I’ve been really enjoying this feature. Not only have I moved my desktop in to my server room and started remoting in using Blast, but I can think of many use cases for this (machines shops, sharing software licenses, etc.).

I’ve had numerous discussions with customers of mine who also say they see tremendous value in this after I brought it to their attention. I’ll update this post later on once I hear back about how some of my customers have deployed it.

3D Acceleration

One thing that is really cool, is the fact that 3D acceleration is enabled and working if the computer has a GPU installed (along with drivers). And no, you don’t need a fancy enterprise GPU. In my setup I’m running a GeForce 550 GTX TI, and a GeForce 640.

Horizon 3D Acceleration dxdiag
Horizon 3D Acceleration Enabled via dxdiag

While 3D acceleration is working, I have to note that the h.264 encoding for the Blast Extreme session is still being handled by the CPU. So while you are getting some great 3D accelerated graphics, depending on your CPU and screen resolution, you may be noticing some choppiness. If you have a higher end CPU, you should be able to get some pretty high resolutions. I’m currently running 2 displays at 1920×1080 on an extremely old Core 2 Quad processor.

H.264 encoding

I spent some time trying to enable the hardware h.264 encoder on the GPUs. Even when using the “NvFBCEnable.exe” (located in C:\Program Files\VMware\VMware Blast\) application to enable hardware encoding, I still notice that the encoding is being done on the CPU. I’m REALLY hoping they change this in future releases.

Hacks?

Another concept that this opens the door for is consumer GPUs providing 3D acceleration without all the driver issues. Technically you could use the CPU settings (to hide the fact the VM is being virtualized), and then install the Horizon Agent as a physical PC (even though it’s being virtualized). This should allow you to use the GPU that you’re passing through, but you still won’t get h.264 encoding on the GPU. This should stop the pesky black screen issue that’s normally seen when using this work around.

Bugs

Also, on a final note… I did find a bug where if any of the physical PCs are powered down or unavailable on the network, any logins from users entitled to that pool will time out and not work. When this issue occurs, a WoL packet is sent to the desktop during login, and the login will freeze until the physical PC becomes available. This occurs during the login phase, and will happen even if you don’t plan on using that pool. More information can be found here:
https://www.stephenwagner.com/2019/03/19/vmware-horizon-view-stuck-authenticating-logging-in/

Apr 272019
 
QuickBooks Online Logo

We all want to make sure our accounting (or clients accounting) data is always backed up, and available for restore in the event of data loss, corruption, or damage. Using QuickBooks desktop, this was easy as you could either use Windows Volume Shadow Services along with your backup and disaster recovery infrastructure, or manual/automatic backups inside of the QuickBooks Desktop application itself.

We traditionally saw IT use the former, and accounting staff using the latter.

Many companies have since migrated from QuickBooks Desktop to QuickBooks online, in an effort to utilize simpler multi-user access, access to your books from anywhere, features and functionality that only come with QBO, app integration and more.

I was one of those people running an extremely old version of QuickBooks, and I needed to upgrade. To keep costs low, and take advantage of some app integration that were available, I decided to upgrade to QuickBooks online.

Overall the upgrade went smooth, however I wanted to make sure that I could backup my data now that I’ve fully migrated over, this is where things got complicated…

QuickBooks Online Protection

Looking through QuickBooks online for some type of backup or export feature, I was unable to find one. I decided to do a google search, and came up with a few documents.

https://community.intuit.com/articles/1779217-do-i-need-to-back-up-my-data-with-quickbooks-online

The above link explains Intuit’s backup and disaster recovery strategy in simple terms for customers. A screenshot below:

According to this page, Intuit backs up our data everynight, stores it on 2 hard drives every change, and copy all data to a third-party hard drive (assuming to take it off-site).

So we know something is being done for protection… It’s a bit scary that a third-party has a copy of the data, but we probably agreed to this when accepting the EULA. There’s still nothing about how we can backup our data.

QuickBooks Online Backup and Export

Doing some further searching, I came across another article.

https://quickbooks.intuit.com/community/Help-Articles/Export-data-file-from-QuickBooks-Online-to-Desktop/td-p/186302

Now we’re talking! According to this article, you can use the data export tool to move your QuickBooks Online data and information to another version of QuickBooks. You could use this not only as a means to export the data, but also to have a backup/copy for yourself (for pease of mind).

I started to follow the instructions in this document, however I quickly noticed that some options that should be available, didn’t show up in my QuickBooks Online interface.

It appeared that I was missing the option to “Download Company Data” under “Moving to QuickBooks Desktop”. I could only export Reports and Lists.

I was wondering if something was wrong with my account because it was brand new, or if I had to enable this feature somewhere.

It turns out that if you are a Canadian QuickBooks Online customer, you do not have access to export or backup your file. You can only export Reports and Lists to Excel. The QuickBooks Online company file export feature is only available on the QuickBooks Online USA (American) version.

I was later able to confirm this by finding this post: https://quickbooks.intuit.com/community/Install/Converting-Quickbooks-online-to-desktop/td-p/182689

So as the individual from the QuickBooks Team stated, this feature is unavailable for QuickBooks Canada.

Worst case scenario, if you are moving from QBO (QuickBooks Online) to QuickBooks desktop, you can find the extremely complicated procedure below, but I wouldn’t recommend it to be used as any type of backup:
https://quickbooks.intuit.com/community/Help-Articles/Export-data-file-from-QuickBooks-Online-to-Desktop/m-p/186302

Hopefully in the future, Intuit will add this functionality to QuickBooks Online Canadian version. Until then, you might be able to try some 3rd party apps, but I haven’t tested or verified any of these.

Comment with your thoughts below!

Apr 242019
 
QB Error 6123

During an upgrade from an extremely old version of QuickBooks to a newer version of QuickBooks, opening the file to upgrade may fail with the following error message:

Error: -6123,0
We're sorry, but QuickBooks can't open your company file.

This specifically occurred with me when I was upgrading from QuickBooks 2008 to Quickbooks 2019 (trial), so I could then migrate my datafile to QuickBooks Online.

QB Error 6123

The Problem

During most (if not all) upgrades and updates, Intuit makes changes to the datafile format for new features, optimizations, or bug fixes. Sometimes these changes can actually be very drastic and intensive, especially if you have a large datafile. Keep in mind that forms and templates may also require upgrades or changes.

When jumping multiple versions, all of these changes must occur in a single swipe.

To make this problem even worse, I believe that there are major (massive, larger than usual) changes done every few upgrades.

Troubleshooting

To troubleshoot this problem, I followed all of QuickBooks documentation on the process but didn’t have any luck. I tried:

  • Upgrading the datafile directly
  • Removing it off the network and upgrading directly
  • Creating a backup using the older version, and restoring with the newer version
  • Using the “Restore older version from backup” feature in “Utilities”

All of these had no effect.

Solution

I finally came across an article that talked about restoring backups and upgrade issues, and as a last attempt recommended creating a portable file on the old version and restoring it to the new version.

Keep in mind that customization will be lost, such as Invoice formats, templates, and other things.

I went ahead and tried this, and to my surprise it worked. I was able to upgrade from QuickBooks 2008 to QuickBooks 2019, and was then able to export and import the datafile in to QuickBooks online.

My 2008 datafile was ~48MB, and took around an hour to convert (I was actually surprised at first, but realized I was performing 11 versions of upgrades to the file).

As always you should confirm that the data is valid. I compared my chart of accounts, profit and loss statements, and trial balance reports, from the old to the new, and all looked good!

Mar 192019
 
VMware Horizon View Logo

I noticed after upgrading to VMware Horizon View 7.8 and VMware Unified Access Gateway 3.5, when attempting to log in to a VMware Horizon View Connection Server via the Horizon Client, I would get stuck on “Authenticating”. If using the HTML client, it would get stuck on “Logging in”.

This will either timeout, or eventually (after numerous minutes) finally load. This occurs both with standard authentication, as well as 2FA/MFA/RADIUS authentication.

The Problem

Originally, I thought this issue was related to 2FA and/or RADIUS, however after disabling both, the issue was still present. In the VDM debug logs, you may find something similar to below:

2019-03-19T16:07:44.971-06:00 INFO  (1064-181C)  UnManagedMachineInformation Wake-on-LAN packet sent to machine comp.domain.com

2019-03-19T16:07:34.296-06:00 INFO (1064-17F0) UnManagedMachineInformation wait ended for startup update, returning false
2019-03-19T16:07:34.296-06:00 INFO (1064-17F0) UnManagedMachineInformation Could not wake up PM comp.domain.com within timeout

The Fix

The apparent delay “Authenticating” or “Logging In” is caused by a Wake On LAN packet being sent to an unmanaged physical workstation that has the VMware View Agent installed. This is occurring because the system is powered off.

After powering on all unmanaged View agents running on physical computers, the issue should be resolved.

Feb 192019
 

Upgrading to Exchange 2016 CU12 may fail when using Let’s Encrypt SSL Certificates

On a Microsoft Exchange 2016 Server, utilizing Let’s Encrypt SSL Certificates, an upgrade to Cumulative Update 12 may fail. This is due to security permissions on the SSL certificate.

I later noted that this occurs on all cumulative updates when using the Let’s Encrypt SSL certificates. This includes Exchange 2016 CU13.

The CU install will fail, some services may function, but the server will not accept e-mail, or allow connections from Microsoft Outlook, or ActiveSync devices. PowerShell and EAC will not function.

The issue can be identified on this failure log:

[02/18/2019 19:24:28.0862] [2] Beginning processing Install-AuthCertificate
[02/18/2019 19:24:28.0867] [2] Ending processing Install-AuthCertificate
[02/18/2019 19:24:28.0868] [1] The following 1 error(s) occurred during task execution:
[02/18/2019 19:24:28.0868] [1] 0. ErrorRecord: Could not grant Network Service access to the certificate with thumbprint XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX because a cryptographic exception was thrown.
[02/18/2019 19:24:28.0868] [1] 0. ErrorRecord: Microsoft.Exchange.Management.SystemConfigurationTasks.AddAccessRuleCryptographicException: Could not grant Network Service access to the certificate with thumbprint XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX because a cryptographic exception was thrown. ---> System.Security.Cryptography.CryptographicException: Access is denied.
at Microsoft.Exchange.Security.Cryptography.X509Certificates.TlsCertificateInfo.CAPIAddAccessRule(X509Certificate2 certificate, AccessRule rule)
at Microsoft.Exchange.Security.Cryptography.X509Certificates.TlsCertificateInfo.AddAccessRule(X509Certificate2 certificate, AccessRule rule)
at Microsoft.Exchange.Management.SystemConfigurationTasks.ManageExchangeCertificate.EnableForServices(X509Certificate2 cert, AllowedServices services, String websiteName, Boolean requireSsl, ITopologyConfigurationSession dataSession, Server server, List`1 warningList, Boolean allowConfirmation, Boolean forceNetworkService)
--- End of inner exception stack trace ---
at Microsoft.Exchange.Configuration.Tasks.Task.ThrowError(Exception exception, ErrorCategory errorCategory, Object target, String helpUrl)
at Microsoft.Exchange.Configuration.Tasks.Task.WriteError(Exception exception, ErrorCategory category, Object target)
at Microsoft.Exchange.Management.SystemConfigurationTasks.InstallExchangeCertificate.EnableForServices(X509Certificate2 cert, AllowedServices services)
at Microsoft.Exchange.Management.SystemConfigurationTasks.InstallExchangeCertificate.InternalProcessRecord()
at Microsoft.Exchange.Configuration.Tasks.Task.b__91_1()
at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePipelineIfFailed)
[02/18/2019 19:24:28.0883] [1] [ERROR] The following error was generated when "$error.Clear();
Install-ExchangeCertificate -services "IIS, POP, IMAP" -DomainController $RoleDomainController
if ($RoleIsDatacenter -ne $true -And $RoleIsPartnerHosted -ne $true)
{
Install-AuthCertificate -DomainController $RoleDomainController
}
" was run: "Microsoft.Exchange.Management.SystemConfigurationTasks.AddAccessRuleCryptographicException: Could not grant Network Service access to the certificate with thumbprint XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX because a cryptographic exception was thrown. ---> System.Security.Cryptography.CryptographicException: Access is denied.
at Microsoft.Exchange.Security.Cryptography.X509Certificates.TlsCertificateInfo.CAPIAddAccessRule(X509Certificate2 certificate, AccessRule rule)
at Microsoft.Exchange.Security.Cryptography.X509Certificates.TlsCertificateInfo.AddAccessRule(X509Certificate2 certificate, AccessRule rule)
at Microsoft.Exchange.Management.SystemConfigurationTasks.ManageExchangeCertificate.EnableForServices(X509Certificate2 cert, AllowedServices services, String websiteName, Boolean requireSsl, ITopologyConfigurationSession dataSession, Server server, List`1 warningList, Boolean allowConfirmation, Boolean forceNetworkService)
--- End of inner exception stack trace ---
at Microsoft.Exchange.Configuration.Tasks.Task.ThrowError(Exception exception, ErrorCategory errorCategory, Object target, String helpUrl)
at Microsoft.Exchange.Configuration.Tasks.Task.WriteError(Exception exception, ErrorCategory category, Object target)
at Microsoft.Exchange.Management.SystemConfigurationTasks.InstallExchangeCertificate.EnableForServices(X509Certificate2 cert, AllowedServices services)
at Microsoft.Exchange.Management.SystemConfigurationTasks.InstallExchangeCertificate.InternalProcessRecord()
at Microsoft.Exchange.Configuration.Tasks.Task.b__91_1()
at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePipelineIfFailed)".
[02/18/2019 19:24:28.0883] [1] [ERROR] Could not grant Network Service access to the certificate with thumbprint XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX because a cryptographic exception was thrown.
[02/18/2019 19:24:28.0883] [1] [ERROR] Access is denied.
[02/18/2019 19:24:28.0883] [1] [ERROR-REFERENCE] Id=CafeComponent___ece23aa8c6744163B617570021d78090 Component=EXCHANGE14:\Current\Release\Shared\Datacenter\Setup
[02/18/2019 19:24:28.0895] [1] Setup is stopping now because of one or more critical errors.
[02/18/2019 19:24:28.0895] [1] Finished executing component tasks.
[02/18/2019 19:24:28.0925] [1] Ending processing Install-CafeRole
[02/18/2019 19:35:09.0688] [0] CurrentResult setupbase.maincore:396: 0
[02/18/2019 19:35:09.0689] [0] End of Setup

The Fix

Unfortunately because Exchange is not working, you won’t be able to use Powershell or the EAC to configure SSL certs.

To resolve this, open up the IIS Manager, right click on the Exchange Web Site, click “Edit Bindings”

IIS Exchange Edit Bindings
IIS Exchange Edit Bindings

Once the “Edit Bindings” windows is open, you’ll want to open BOTH https bindings, and click “Edit”, and then change the SSL Certificate from the Let’s Encrypt SSL cert, to the self-signed Exchange certificate that ships on the brand new install. The self-signed certification most likely will be labelled as the computer name.

Exchange SSL Bindings
Exchange SSL Bindings

If you configured the Let’s Encrypt SSL certificate on the “Exchange Backend” IIS site, you’ll also need to repeat these steps on that as well.

You can now restart the server, run the “setup.exe” on CU12 again, and it will attempt to continue and repair Exchange 2016 Cumulative Update 12.

Final Note

After the update is complete, you’ll want to restart the server. You’ll notice that the acme script, whether run automatically or manually, will not set the Let’s Encrypt certificate up again (because it’s not due for renewal). You’ll need to run the letsencrypt.exe file, and force an auto renewal which will kick off the Exchange configuration scripts (or you can manually set the certificate if you’re comfortable applying Exchange SSL certificates via PowerShell.

Feb 182019
 
ESXi Fatal error: 8 (Device Error)

Unable to boot ESXi from USB or SD Card on HPe Proliant Server

After installing HPe iLO Amplifier on your network and updating iLO 4 firmware to 2.60 or 2.61, you may notice that your HPe Proliant Servers may fail to boot ESXi from a USB drive or SD-Card.

This was occuring on 2 ESXi Hosts. Both were HPe Proliant DL360p Gen8 Servers. One server was using an internal USB drive for ESXi, while the other was using an HPe branded SD Card.

The issue started occuring on both hosts after a planned InfoSight implementation. Both hosts iLO controllers firmware were upgraded to 2.61, iLO Amplifier was deployed (and the servers added), and the amplifier was connected to an HPe InfoSight account.

Update – May 24th 2019: As an HPe partner, I have been working with HPe, the product manager, and development team on this issue. HPe has provided me with a fix to test that I have been able to verify fully resolves this issue! Stay tuned for more information!

Update – June 5th 2019: Great news! As Bob Perugini (WW Product Manager at HPe) put it: “HPE is happy to announce that this issue has been fixed in latest version of iLO Amplifier Pack, v1.40. To download iLO Amplifier Pack v1.40, go to http://www.hpe.com/servers/iloamplifierpack and click “download”.” Scroll to the bottom of the post for more information!

Please see below for errors:

Errors

ESXi Fatal error: 8 (Device Error)
ESXi Fatal error: 8 (Device Error)
Error loading /s.v00
Compressed MD5: 00000000000000000000000000
Decompressed MD5: 00000000000000000000000000
Fatal error: 8 (Device Error)
Error mboot.c32 attempted DOS system call
Error mboot.c32 attempted DOS system call
mboot.c32: attempted DOS system call INT 21 0d00 E8004391
boot:

Symptoms

This issue may occur intermittently, on the majority of boots, or on all boots. Re-installing ESXi on the media, as well as replacing the USB/SD Card has no effect. Installation will be successful, however you the issue is still experiences on boot.

HPe technical support was unable to determine the root of the issue. We found the source of the issue and reported it to HPe technical support and are waiting for an update.

The Issue and Fix

This issue occurs because the HPe iLO Amplifier is running continuous server inventory scans while the hosts are booting. When one inventory completes, it restarts another scan.

The following can be noted:

  • iLO Amplifier inventory percentage resets back to 0% and starts again numerous times during the server boot
  • Inventory scan completes, only to restart again numerous times during the server boot
  • Inventory scan resets back to 0% during numerous different phases of BIOS initialization and POST.
HPe iLO Amplifier Inventory
HPe iLO Amplifier Inventory

We noticed that once the HPe iLO Amplifier Virtual Machine was powered off, not only did the servers boot faster, but they also booted 100% succesfully each time. Powering on the iLO Amplifier would cause the ESXi hosts to fail to boot once again.

I’d also like to note that on the host using the SD-Card, the failed boot would actually completely lock up iLO, and would require physical intervention to disconnect and reconnect the power to the server. We were unable to restart the server once it froze (this did not happen to the host using the USB drive).

There are some settings on the HPe iLO amplifier to control performance and intervals of inventory scans, however we noticed that modifying these settings did not alter or stop the issue, and had no effect.

As a temporary workaround, make sure your iLO amplifier is powered off during any maintenance to avoid hosts freezing/failing to boot.

To fully resolve this issue, upgrade your iLO Amplifier to the latest version (1.40 as of the time of this update). The latest version can be downloaded at: http://www.hpe.com/servers/iloamplifierpack.

Update – April 10th 2019

I’ve attempted to try downgrading to the earliest supported iLo version 2.54, and the issue still occurs.

I also upgraded to the newest version 2.62 which presented some new issues.

On the first boot, the BIOS reported memory access issues on Processor 1 socket 1, then another error reporting memory access issues on Processor 1 socket 4.

I disconnected the power cables, reconnected, and restarted the server. This boot, the server didn’t even detect the bootable USB stick.

Again, after shutting down the iLo Amplifier, the server booted properly and the issue disappeared.

Update – May 24th 2019

As an HPe partner, I have been working with HPe, the product manager, and development team on this issue. HPe has provided me with a fix to test that I have been able to verify fully resolves this issue! Stay tuned for more information!

Update – June 5th 2019 – ITS FIXED!!!

Great news as the issue is now fixed! As Bob Perugini (WorldWide Product Manager at HPe) said it:

HPE is happy to announce that this issue has been fixed in latest version of iLO Amplifier Pack, v1.40.

To download iLO Amplifier Pack v1.40, go to http://www.hpe.com/servers/iloamplifierpack and click “download”.


Here’s what’s new in iLO Amplifier Pack v1.40:
─ Available as a VMware ESXi appliance and as a Hyper-V appliance (Hyper-V is new)
─ VMware tools have been added to the ESXi appliance
─ Ability to schedule the time of the daily transmission of Active Health System (AHS) data to InfoSight
─ Ability to opt-in and allow the IP address and hostname of the server to be transmitted to InfoSight and displayed
─ Test connectivity button to help verify iLO Amplifier Pack has successfully connected to InfoSight
─ Allow user authentication credentials for the proxy server when connecting to InfoSight
─ Added ability to specify IP address or hostname for the HPE RDA connection when connection to InfoSight
─ Ability to send updated AHS data “now” for an individual server
─ Ability to stage firmware and driver updates to the iLO Repository and then deploy the staged updates at a later date or time (HPE Gen10 servers only)
─ Allow the firmware and driver updates of servers whose iLO has been configured in CNSA (Commercial National Security Algorithm) mode (HPE Gen10 servers only)