Jun 192022
 
VMware vSphere 7 Logo

We all know that vMotion is awesome, but what is even more awesome? Optimizing VMware vMotion to make it redundant and faster!

vMotion allows us to migrate live Virtual Machines from one ESXi host to another without any downtime. This allows us to perform physical maintenance on the ESXi hosts, update and restart the hosts, and also load balance VMs across the hosts. We can even take this a step further use DRS (Distributed Resource Scheduler) automation to intelligently load the hosts on VM boot and to dynamically load balance the VMs as they run.

Picture of VMware vMotion diagram
VMware vMotion

In this post, I’m hoping to provide information on how to fully optimize and use vMotion to it’s full potential.

VMware vMotion

Most of you are probably running vMotion in your environment, whether it’s a homelab, dev environment, or production environment.

I typically see vMotion deployed on the existing data network in smaller environments, I see it deployed on it’s own network in larger environments, and in very highly configured environments I see it being used with the vMotion TCP stack.

While you can preform a vMotion with 1Gb networking, you certainly almost always want at least 10Gb networking for the vMotion network, to avoid any long running VMs. Typically most IT admins are happy with live migration vMotion’s in the seconds, and not the minutes.

VMware vMotion Optimization

So you might ask, if vMotion is working and you’re satisfied, what is there to optimize? There’s actually a few things, but first let’s talk about what we can improve on.

We’re aiming for improvements with:

  • Throughput/Speed
    • Faster vMotion
      • Faster Speed
      • Less Time
    • Migrate more VMs
      • Evacuate hosts faster
      • Enable more aggressive DRS
      • Migrate many VMs at once very quickly
  • Redundancy
    • Redundant vMotion Interfaces (NICs and Uplinks)
  • More Complex vMotion Configurations
    • vMotion over different subnets and VLANs
      • vMotion routed over Layer 3 networks

To achieve the above, we can focus on the following optimizations:

  1. Enable Jumbo Frames
  2. Saturation of NIC/Uplink for vMotion
  3. Multi-NIC/Uplink vMotion
  4. Use of the vMotion TCP Stack

Let’s get to it!

Enable Jumbo Frames

I can’t stress enough how important it is to use Jumbo Frames for specialized network traffic on high speed network links. I highly recommend you enable Jumbo Frames on your vMotion network.

Note, that you’ll need to have a physical switch and NICs that supports Jumbo frames.

In my own high throughput testing on a 10Gb link, without using Jumbo frames I was only able to achieve transfer speeds of ~6.7Gbps, whereas enabling Jumbo Frames allowed me to achieve speeds of ~9.8Gbps.

When enabling this inside of vSphere and/or ESXi, you’ll need to make sure you change and update the applicable vmk adapter, vSwitch/vDS switches, and port groups. Additionally as mentioned above you’ll need to enable it on your physical switches.

You may assume that once you configure a vMotion enabled NIC, that when performing migrations you will be able to fully saturate it. This is not necessarily the case!

When performing a vMotion, the vmk adapter is bound to a single thread (or CPU core). Depending on the power of your processor and the speed of the NIC, you may not actually be able to fully saturate a single 10Gb uplink.

In my own testing in my homelab, I needed to have a total of 2 VMK adapters to saturate a single 10Gb link.

If you’re running 40Gb or even 100Gb, you definitely want to look at adding multiple VMK adapters to your vMotion network to be able to fully saturate a single NIC or Uplink.

You can do this by simply configuring multiple VMK adapters per host with different IP addresses on the same subnet.

One important thing to mention is that if you have multiple physical NICs and Uplinks connected to your vMotion switch, this change will not help you utilize multiple physical interfaces (NICs/Uplinks). See “Multi-NIC/Uplink vMotion”.

Please note: As of VMware vSphere 7 Update 2, the above is not required as vMotion has been optimized to use multiple streams to fully saturate the interface. See VMware’s blog post “Faster vMotion Makes Balancing Workloads Invisible” for more information.

Multi-NIC/Uplink vMotion

Another situation is where we may want to utilize multiple NICs and Uplinks for vMotion. When implemented correctly, this can provide load balancing (additional throughput) as well as redundancy on the vMotion network.

If you were to simply add additional NIC interfaces as Uplinks to your vMotion network, this would add redundancy in the event of a failover but it wouldn’t actually result in increased speed or throughput as special configuration is required.

To take advantage of the additional bandwidth made available by additional Uplinks, we need to specially configure multiple portgroups on the switch (vSwitch or vDS Distributed Switch), and configure each portgroup to only use one of the Uplinks as the “Active Uplink” with the rest of the uplinks under “Standby Uplink”.

Example Configuration

  • vSwitch or vDS Switch
    • Portgroup 1
      • Active Uplink: Uplink 1
      • Standby Uplinks: Uplink 2, Uplink 3, Uplink 4
    • Portgroup 2
      • Active Uplink: Uplink 2
      • Standby Uplinks: Uplink 1, Uplink 3, Uplink 4
    • Portgroup 3
      • Active Uplink: Uplink 3
      • Standby Uplinks: Uplink 1, Uplink 2, Uplink 4
    • Portgroup 4
      • Active Uplink: Uplink 4
      • Standby Uplinks: Uplink 1, Uplink 2, Uplink 3

You would then place a single or multiple vmk adapters on each of the portgroups per host, which would result in essentially mapping the vmk(s) to the specific uplink. This will allow you to utilize multiple NICs for vMotion.

And remember, you may not be able to fully saturate a NIC interface (as stated above) with a single vmk adapter, so I highly recommend creating multiple vmk adapters on each of the Port groups above to make sure that you’re not only using multiple NICs, but that you can also fully saturate each of the NICs.

For more information, see VMware’s KB “Multiple-NIC vMotion in vSphere (2007467)“.

Use of the vMotion TCP Stack

VMware released the vMotion TCP Stack to provided added security to vMotion capabilities, as well as introduce vMotion over multiple subnets (routed vMotion over layer 3).

Using the vMotion TCP Stack, you can isolate and have the vMotion network using it’s own gateway separate from the other vmk adapters using the traditional TCP stack on the ESXi host.

This stack is optimized for vMotion.

Please note, that troubleshooting processes may be different when Troubleshooting vMotion using the vMotion TCP/IP Stack (click the link for my blog post on troubleshooting).

For more information, see VMware’s Documentation on “vMotion TCP/IP Stack“.

Additional resources:

VMware – How to Tune vMotion for Lower Migration Times?

Jun 182022
 
Nvidia GRID Logo

When performing a VMware vMotion on a Virtual Machine with an NVIDIA vGPU attached to it, the VM may freeze during migration. Additionally, when performing a vMotion on a VM without a vGPU, the VM does not freeze during migration.

So why is it that adding a vGPU to a VM causes it to become frozen during vMotion? This is referred to as the VM Stun Time.

I’m going to explain why this happens, and what you can do to reduce these STUN times.

VMware vMotion

First, let’s start with traditional vMotion without a vGPU attached.

VMware vMotion with vSphere and ESXi
VMware vMotion with vSphere

vMotion allows us to live migrate a Virtual Machine instance from one ESXi host, to another, with (visibly) no downtime. You’ll notice that I put “visibly” in brackets…

When performing a vMotion, vSphere will migrate the VM’s memory from the source to destination host and create checkpoints. It will then continue to copy memory deltas including changes blocks after the initial copy.

Essentially vMotion copies the memory of the instance, then initiates more copies to copy over the changes after the original transfer was completed, until the point where it’s all copied and the instance is now running on the destination host.

VMware vMotion with vGPU

For some time, we have had the ability to perform a vMotion with a VM that as a GPU attached to it.

VMware vSphere with NVIDIA vGPU
VMware VMs with vGPU

However, in this situation things work slightly different. When performing a vMotion, it’s not only the system RAM memory that needs to be transferred, but the GPU’s memory (VRAM) as well.

Unfortunately the checkpoint/delta transfer technology that’s used with then system RAM isn’t available to transfer the GPU, which means that the VM has to be stunned (frozen) to stop it so that the video RAM can be transferred and then the instance can be initialized on the destination host.

STUN Time

The STUN time is essentially the time it takes to transfer the video RAM (framebuffer) from one host to another.

When researching this, you may find examples of the time it takes to transfer various sizes of VRAM. An example would be from VMware’s documentation “Using vMotion to Migrate vGPU Virtual Machines“:

NVIDIA vGPU Estimated STUN Times
Expected STUN Times for vMotion with vGPU on 10Gig vMotion NIC

However, it will always vary depending on a number of factors. These factors include:

  • vMotion Network Speed
  • vMotion Network Optimization
    • Multi-NIC vMotion to utilize multiple NICs
    • Multi-vmk vMotion to optimize and saturate single NICs
  • Server Load
  • Network Throughput
  • The number of VM’s that are currently being migrated with vMotion

As you can see, there’s a number of things that play in to this. If you have a single 10Gig link for vMotion and you’re migrating many VMs with a vGPU, it’s obviously going to take longer than if you were just migrating a single VM with a vGPU.

Optimizing and Minimizing vGPU STUN Time

There’s a number of things we can look at to minimize the vGPU STUN times. This includes:

  • Upgrading networking throughput with faster NICs
  • Optimizing vMotion (Configure multiple vMotion VMK adapters to saturate a NIC)
  • Configure Multi-NIC vMotion (Utilize multiple physical NICs to increase vMotion throughput)
  • Reduce DRS aggressiveness
  • Migrate fewer VMs at the same time

All of the above can be implemented together, which I would actually recommend.

In short, the faster we migrate the VM, the less the STUN Time will be. Check out my blog post on Optimizing VMware vMotion which includes how to perform the above recommendations.

Hope this helps!

Mar 062022
 
Azure AD SSO with Horizon

Whether deploying VDI for the first time or troubleshooting existing Azure AD SSO issues for an existing environment, special consideration must be made for Microsoft Azure AD SSO and VDI.

When you implement and use Microsoft 365 and Office 365 in a VDI environment, you should have your environment configured to handle Azure AD SSO for a seamless user experience, and to avoid numerous login prompts when accessing these services.

Microsoft Azure Active Directory has two different methods for handling SSO (Single Sign On), these include SSO via a Primary Refresh Token (PRT) and Azure Seamless SSO. In this post, I’ll explain the differences, and when to use which one.

Microsoft Azure AD SSO and VDI

What does Azure AD SSO do?

Azure AD SSO allows your domain joined Windows workstations (and Windows Servers) to have a Single Sign On experience so that users can have an single sign-on integrated experience when accessing Microsoft 365 and/or Office 365.

When Azure AD SSO is enabled and functioning, your users will not be prompted nor have to log on to Microsoft 365 or Office 365 applications or services (including web services) as all this will be handled transparently in the background with Azure AD SSO.

For VDI environments, especially non-persistent VDI (VMware Instant Clones), this is an important function so that users are not prompted to login every time they launch an Office 365 application.

Persistent VDI is not complex and doesn’t have any special considerations for Azure AD SSO, as it will function the same way as traditional workstations, however non-persistent VDI requires special planning.

Please Note: Organizations often associate the Office 365 login prompts to activation issues when in fact activation is functioning fine, however Azure AD SSO is either not enabled, incorrect configured, or not functioning which is why the users are being prompted for login credentials every time they establish a new session with non-persistent VDI. After reading this guide, it should allow you to resolve the issue of Office 365 login prompts on VDI non-persistent and Instant Clone VMs.

Azure AD SSO methods

There are two different ways to perform Azure AD SSO in an environment that is not using ADFS. These are:

  • Azure AD SSO via Primary Refresh Token
  • Azure AD Seamless SSO

Both accomplish the same task, but were created at different times, have different purposes, and are used for different scenarios. We’ll explore this below so that you can understand how each works.

Screenshot of a Hybrid Azure AD Joined login
Hybrid Azure AD Joined Login

Fun fact: You can have both Azure AD SSO via PRT and Azure AD Seamless SSO configured at the same time to service your Active Directory domain, devices, and users.

Azure SSO via Primary Refresh Token

When using Azure SSO via Primary Refresh Token, SSO requests are performed by Windows Workstations (or Windows Servers), that are Hybrid Azure AD Joined. When a device is Hybrid Azure AD Joined, it is joined both to your on-premise Active Directory domain, as well registered to your Azure Active Directory.

Azure SSO via Primary Refresh token requires the Windows instance to be running Windows 10 (or later), and/or Windows Server 2016 (or later), as well the Windows instance has to be Azure Hybrid AD joined. If you meet these requirements, SSO with PRT will be performed transparently in the background.

If you require your non-persistent VDI VMs to be Hybrid Azure AD joined and require Azure AD SSO with PRT, special considerations and steps are required:

This includes:

  • Scripts to automatically unjoin non-persistent (Instant Clone) VDI VMs from Azure AD on logoff.
  • Scripts to cleanup old entries on Azure AD

If you properly deploy this, it should function. If you don’t require your non-persistent VDI VMs to be Hybrid Azure AD joined, then Azure AD Seamless SSO may be better for your environment.

Azure AD Seamless SSO

Microsoft Azure AD Seamless SSO after configured and implemented, handles Azure AD SSO requests without the requirement of the device being Hybrid Azure AD joined.

Seamless SSO works on Windows instances instances running Windows 7 (or later, including Windows 10 and Windows 11), and does NOT require the the device to be Hybrid joined.

Seamless SSO allows your Windows instances to access Azure related services (such as Microsoft 365 and Office 365) and provides a single sign-on experience.

This may be the easier method to use when deploying non-persistent VDI (VMware Instant Clones), if you want to implement SSO with Azure, but do not have the requirement of Hybrid AD joining your devices.

Additionally, by using Seamless SSO, you do not need to implement the require log-off and maintenance scripts mentioned in the above section (for Azure AD SSO via PRT).

To use Azure AD Seamless SSO with non-persistent VDI, you must configure and implement Seamless SSO, as well as perform one of the following to make sure your devices do not attempt to Hybrid AD join:

  • Exclude the non-persistent VDI computer OU containers from Azure AD Connect synchronization to Azure AD
  • Implement a registry key on your non-persistent (Instant Clone) golden image, to disable Hybrid Azure AD joining.

To disable Hybrid Azure AD join on Windows, create the registry key on your Windows image below:

HKLM\SOFTWARE\Policies\Microsoft\Windows\WorkplaceJoin: "BlockAADWorkplaceJoin"=dword:00000001

Conclusion

Different methods can be used to implement SSO with Active Directory and Azure AD as stated above. Use the method that will be the easiest to maintain and provide support for the applications and services you need to access. And remember, you can also implement and use both methods in your environment!

After configuring Azure AD SSO, you’ll still be required to implement the relevant GPOs to configure Microsoft 365 and Office 365 behavior in your environment.

Additional Resources

Please see below for additional information and resources:

Jan 212022
 
sconfig Server Configuration menu

We’re all used to updating our Windows Server operating systems with the Windows Update GUI, but did you know that you can update your server using command prompt and “sconfig”?

The past few years I’ve been managing quite a few Windows Server Core Instances that as we all know, do not have a GUI. In order to update those instances, you need to run Windows Update using the command line, but this method actually also works on normal Windows Server instances with the GUI as well!

Windows Update from CLI (Command Prompt)

Please enjoy this video or read on for why and how!

Why?

Using a GUI is great, however sometimes it’s not needed, and sometimes it even causes problems if it looses the backend connection where it’s pulling the data from. I’ve seen this true on newer Windows operating systems where the Windows Update GUI stops updating and you just sit there thinking the updates are running, when they are actually all complete.

The GUI also creates additional overhead and clutter. If there was an easier alternative to perform this function, wouldn’t it just make sense?

On Windows Server instances that have a GUI, I find it way faster and more responsive to just open an elevated (Administrative) command prompt, and kick off Windows Updates from there.

How

You can use this method on all modern Windows Server versions:

  • Windows Server (with a GUI)
  • Windows Server Core (without a GUI)

This also works with Windows Server Update Services so you can use this method either connecting to Windows Update (Microsoft Update) or Windows Server Update Services (WSUS).

Now lets get started!

  1. Open an Administrative (elevated) command prompt
  2. Run “sconfig” to launch the “Server Configuration” application
    command prompt launch sconfig
  3. Select option “6” to “Download and Install Windows Updates”
    sconfig Server Configuration menu
  4. Choose “A” for all updates, or “R” for recommended updates, and a scan will start
  5. After the available updates are shown, choose “A” for all updates, “N” for no updates, or “S” for single update selection

After performing the above, the updates will download and install.

sconfig Windows Update running
“sconfig” Windows Update downloading and installing

I find it so much easier to use this method when updating many/multiple servers instead of the GUI. Once the updates are complete and you’re back at the “Server Configuration” application, you can use option “13” to restart Windows.

Jan 162022
 

Welcome to Episode 04 of The Tech Journal Vlog at www.StephenWagner.com

The Tech Journal Vlog Episode 04

In this episode

Updates

  • VMware Horizon
    • Apache Log4j Mitigation with VMware Products
  • Homelab Update
    • HPE MSA 2040 vs Synology DS1621+
    • Migrating from MSA 2040 to a Synology DS1621+
    • Synology Benchmarking NVME Cache
  • DST Root CA X3 Expiration
    • End of Life Operating Systems

New Blog/Video Posts

Life Update/Fun Stuff

  • Work
  • Travel
  • Move

Current Projects

  • Synology DS1621+

Don’t forget to like and subscribe!
Leave a comment, feedback, or suggestions!

Dec 022021
 

As I’m watching the Synology: 2022 AND BEYOND event live I wanted to write this post to share with you some of my favorite parts.

For those of you wanting to tune in (while live) or watch the replay you can view it below.

You can also visit: https://event.synology.com/en-us/annual_event to watch the stream (or replay) and find out more information.

DSM 7.0.1 and DSM 7.1

As an IT professional and enthusiast, here’s a list of some of my favorite new features, functionalities and improvements.

DSM 7.0.1

  • High Availability Improvements
    • 40% less time needed for failovers and switchovers
    • 80% less time needed for system updates
  • System Responsiveness and Performance enhancements
    • 56x faster SMB (Windows File Share) searches
    • 70% speed up in application launches
    • 80% faster RAID 6 performance (for double parity RAID)
  • Enhanced Drive Replacement
    • Replace predicted failure/presumed unhealthy/to-be-retired drives with unused healthy drives that are in other bays.
    • Switch without degrading your array/storage pool
  • Auto Drive Replacement
    • Clones predicted failure/presumed unhealthy drive to available hot spare, switching automatically when the drive fails without requiring RAID rebuild time
  • Volume-wide data deduplication (on all flash arrays)
    • Scheduled or Manual Volume Wide Deduplication on all flash volumes
    • Click here for information on supported models and configurations
  • Synology Directory Server
    • Secondary domain controllers provide additional redundancy and availability of directory services
    • Provides ability to deploy additional domain controllers at additional locations

DSM 7.1 (Coming Soon)

  • SMB Multichannel
    • Utilize multiple network links and network connections to combine speed with SMB (Windows File Share) file transfers
    • Provides redundancy and fault tolerance
  • DFS Support over SMB
    • Combine multiple Synology NAS file servers to provide file services to your network using Distributed File System (DFS)
  • Active Insight
    • Numerous updates to make Active Insight more powerful when it comes to managing numerous Synology NAS units.
    • Enhanced control and information on System Updates
    • Suspicious activity detection recognizes suspicious activity on the NAS units (example: unusual login times and locations, unused accounts being used after being idle, failed logins, etc).
    • Task monitoring for Hyper Backup
  • Synology Directory
    • Read-only domain controller support provides ability to deploy read only domain controllers at remote sites that may not be physically secure while providing local cached copies of the directory.

And that’s not all…

Synology has also made numerous improvements to their Data Protection Services, Surveillance suite, Wireless Networking, Synology C2 Cloud, and more, but you’ll have to check it out for yourself. There are tons of other goodies for office workers, small business owners, photography enthusiast, and more that I haven’t included in this list.

Dec 022021
 

In a VMware Horizon environment with DUO MFA configured via RADIUS on the VMware Horizon Connection Server, you may notice authentication issues when logging in through a UAG (Unified Access Gateway) after upgrading to VMware Horizon 8 Version 2111.

During this condition, you can still login and use the connection server directly with MFA working, however all UAG connections will get stuck on authenticating.

Horzion 8 Version 2111 UAG Stuck on Authenticating using DUO MFA (RADIUS)

Disabling MFA and/or RADIUS on the connection server will allow the UAG to function, however MFA will be disabled. This occurs on upgrades to version 2111 of the UAG both when configuring fresh, and importing the JSON configuration backup.

Temporary Fix

Update January 26 2022: VMware has now released version 2111.2 of the Unified Access Gateway which resolves this issue. You can download it here, or view the release notes here.

Update January 12 2022: It appears VMware now has a KB on this issue at: https://kb.vmware.com/s/article/87253.

Temporary workaround/fix: To fix this issue, log on to the UAG and under “Horizon Edge Settings”, configure “Client Encryption Mode” to “Disabled”.

“Client Encryption Mode” is a new setting on UAG 2111 (and UAG 2111.1) that enables new functionality. Disabling this reverts the UAG to the previous behavior of older Unified Access Gateway versions.

More information on “Client Encryption Mode” can be found at https://docs.vmware.com/en/Unified-Access-Gateway/2111/uag-deploy-config/GUID-1B8665A2-485E-4471-954E-56DB9BA540E9.html.

Another workaround is to deploy an older version of the UAG, version 2106. After downgrading, the UAG functions with DUO and RADIUS even though the Connection Server is at version 2111.

If you use an older version of the UAG, please make sure that you mitigate against the Apache log4j vulnerabilities on the UAG using information from the following post: https://kb.vmware.com/s/article/87092.

Oct 112021
 
Windows Server 2022 Logo

Today we’re going to discuss and deploy Active Directory Certificate Services on a Windows Server 2022 Server. Additionally, we’ll also be generating a domain certificate request inside of IIS and then assign the resultant certificate to a WSUS Server.

This video will demonstrate and explain the process of deploying a Windows Server 2022 Certification Authority with AD CS.

Check it out and feel free to leave a comment! Scroll down below for more information and details on the guide.

Windows Server 2022: Active Directory Certificate Services Discussion and Installation Guide

Who’s this guide for

This guide is perfect for a seasoned IT professional or a beginner who is looking at getting experience with Windows Server 2022.

What’s included in the video

In this guide I will walk you through the following:

  • Discussion
    • SSL Certificates (Host verification)
    • Internal Root Certification Authorities (Root CAs)
    • Internal Root CA vs Public Trusted Root CAs
    • HTTPS Scanning (Web Filtering) and SSL Certificates
    • Intermediate Certificate Authorities
    • Why ADCS?
    • AD CS Certificate Templates
    • Encryption
    • Certificate Issuance
  • Demonstration
    • Server Manager Role Installation
    • MMC Snap-in for Certificates (Local Computer)
      • Root CAs
    • Install Active Directory Certificate Services (AD CS)
      • Add Server Role
      • Root CA Trust Discussion
      • AD CS Installation on Domain Controller Installation
      • AD CS Prerequisites
      • Web Enrollment Discussion
      • AD CS and IIS Discussion
    • Install Internet Information Services (IIS) as pre-requisite
    • Configure Active Directory Certificate Services (AD CS)
      • Credentials
      • Role Configuration
      • Enterprise CA vs Standalone CA
      • Root CA vs Subordinate CA
      • Private Key Creation and Cryptographic options
      • Root CA Naming
      • Validity Period
    • Certification Authority MMC Usage
    • Root CA Replication to Domain (“gpupdate /force” and restart)
    • AD CS Certificate Templates Overview
      • Certificate Templates MMC
      • Duplicate and Customize Web Server Certificate Template
      • Enable Auto-Enrollment for Certificate Template
    • Use IIS to request certificate from Active Directory Certification Authority
      • Create Domain Certificate
    • Enable SSL on WSUS Server using Active Directory Certificate Services Certificate
      • Bind new certificate to IIS Web Server
      • Update GPO to reflect SSL URL and port number
      • Run “iisreset” on elevated command prompt
    • Demonstration Summary

What’s required

To get started you’ll need:

  • 1 x Server (Virtual Machine or Physical Server)
  • Microsoft Windows Server 2022 Licensing
  • A running Windows Server 2022 Instance (OSE)
  • A network router and/or firewall

Hardware/Software used in this demonstration

  • VMware vSphere
  • HPE DL360p Gen8 Server
  • Microsoft Windows Server 2022
  • pfSense Firewall
Oct 102021
 
VMware vSphere 7 Logo

In this post, I wanted to go over some Backup and Restore tips and tricks when it comes to VMware vCSA Updates and Upgrades.

We’ve almost all been there, performing an update or upgrade of the VMware vCenter Server Appliance when it fails, and we must restore from a backup. There’s also times where the update or upgrade has been successful, however numerous issues occur afterwards prompting for the requirement of a restore from backup.

In this post, I wanted to briefly go over the methods of backups (and restores) for the vCSA, as well as some Tips and Tricks which might help you out for avoiding failed updates or upgrades in the future!

We all want to avoid a failed update or upgrade! 🙂

vCSA Update Installation
vCSA Update Installation

VMware vCSA Update Tips and Tricks for Backup and Restore

Please enjoy this video version of the blog post:

vCSA Update and Upgrade – Tips and Tricks for Backup and Restore

vCSA Backup methods

There are essentially two backup methods for backing up the vCenter Server Appliance:

  1. vCSA Management Interface Backup
  2. vSphere/ESXi Virtual Machine Snapshot

vCSA Management Interface Backup

If you log in to the vCSA Management Interface, you can configure a scheduled backup that will perform a full backup of your vCSA (and vCenter Server) instance.

This backup can be automatically ran and saved to an HTTP, HTTPS, FTP, FTPS, SFTP, NFS, or SMB destination. It’s a no-brainer if you have a Windows File Server or an NFS datastore.

vCSA Backup Screenshot
vCSA Backup

In the event of a failed update/upgrade or a disaster, this backup can be restored to a new vCSA instance to recover from the failure.

For more information on backups from the vCSA Management Interface, please see https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vcenter.install.doc/GUID-8C9D5260-291C-44EB-A79C-BFFF506F2216.html.

For information on restoring a vCSA file based backup, please see https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vcenter.install.doc/GUID-F02AF073-7CFD-45B2-ACC8-DE3B6ED28022.html.

vSphere/ESXi Virtual Machine Snapshot

In addition to the scheduled automatic backups configured above, you should snapshot your vCSA appliance VM prior to initiating an update or upgrade. In the event of a failure, you can easily restore the vCSA VM snapshot to get back to a running state.

vCSA Snapshot Screenshot
vCSA Snapshot

Only after you test and confirm the upgrade or update was successful should you delete the snapshot.

You should also have your Backup application or suite performing regularly snapshot based backups of your vCSA.

Additional Tips and Tricks

I have a few very important tips and tricks to share which may help you either avoid a failed update or upgrade, or increase the chances of a successful restore from backup.

  1. Gracefully Shutdown and Restart the vCSA Appliance before Upgrading
  2. Application Consistent Snapshot – Snapshot after graceful shutdown

Let’s dive in to these below.

Gracefully Shutdown and Restart the vCSA Appliance before Upgrading

I noticed that I significantly reduced the amount of failed upgrades by simply gracefully shutting down and restarting the vCenter Server Appliance prior to an upgrade.

This allows you to clear out the memory, virtual memory, and restart all vCenter services prior to starting the upgrade.

Please Note: Make sure that you give the vCSA appliance enough time to boot, start services, and let some of the maintenance tasks run before initiating an upgrade.

Application Consistent Snapshot – Snapshot after graceful shutdown

Most VMware System Administrators I have talked to, usually snapshot the running vCSA appliance and do not snapshot the memory. This creates a crash consistent snapshot.

If you follow my advice above and gracefully shutdown and restart the vCSA appliance, you can use this time to perform a VM snapshot after a graceful shutdown. This will provide you with an application consistent snapshot instead of a crash consistent snapshot.

If you perform an application consistent snapshot by gracefully shutting down the VM prior to creating the snapshot, the virtual machine and database inside of it will be in a cleaner state.

Conclusion

Some of the Tips and Tricks in this post definitely aren’t necessary, however they can help you increase the chance of a successful upgrade, and a successful restore in the event of a failed upgrade.

For more information on upgrading the vCenter Server Appliance, please visit https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vcenter.upgrade.doc/GUID-30485437-B107-42EC-A0A8-A03334CFC825.html.

Oct 092021
 

One of the new real cool features of the Synology C2 Cloud and Synology DSM 7.0 is the ability to back your Synology DiskStation NAS up to the cloud.

Using the Synology’s “C2 Storage” service, you can use Synology Hyper Backup on DSM 7.0 to backup your your File Shares, NAS, applications, and data to the cloud.

Below, I will show the process of creating a Hyper Backup Backup task to backup a few Synology File Shares to the Synology C2 Storage Service.

NAS Backup (and Restore) with Synology C2 Cloud Video

For a full video guide and demonstration, please enjoy this video! Please continue scrolling for a text guide (with images).

Synology DiskStation NAS Backup (and Restore) with Synology C2 Storage Cloud!

Backup NAS to Synology C2 Storage

It’s very easy to back your NAS up to the cloud. Here’s a quick guide showing how it’s done.

We’re going to assume you already have some File Shares configured on your NAS. These can be Windows File Shares, NFS Exports, and even Active Backup for Business storage repositories. In my case, I want to back up 2 Windows File Shares to the Synology C2 Storage service.

  1. Open Hyper Backup
  2. If you have no existing backup jobs, the “Backup Wizard” should automatically open to configure a new job. If you have existing jobs, select the “+” on the top left of the window.
  3. Under “Backup Destination”, choose “Synology C2 Storage”.
    Hyper Backup Destination
  4. This will open a new window asking you to log on to the Synology C2 Storage Service. You may have to create a new account, or log in with your existing Synology credentials. From here you can start a free trial.
    Synology C2 Storage - Get Free Trial Now
  5. Choose a subscription and plan. You should get a 30-day free trial.
    Synology C2 Storage - Select your C2 subscription
  6. Confirm your subscription
    Synology C2 Storage - Confirm Subscription
  7. Accept the TOS (Terms of Service)
    Synology C2 Storage - Terms of Service
  8. You must now “Grant access to C2 Storage”. This will allow your user account to access the C2 Storage service.
    Synology C2 Storage - Grant Access
  9. You will now be presented with information on your Synology C2 Storage Account
    Synology C2 Storage Account Information
  10. You will now be redirected back to Hyper Backup. You’ll need to create a name for the new backup task destination directory.
    Hyper Backup Destination Settings
  11. Here’s where you will configure which File Shares (or volumes) to backup to Synology C2.
    Hyper Backup Choose Data Backup Sources
  12. You also have the ability to backup Synology Applications. This is helpful should you wish to backup your entire Synology NAS to Synology’s C2 cloud. I’m only doing shares, so I hit next without selecting anything.
    Hyper Backup Application Backup
  13. Now we will configure the Backup Task settings. This includes the task name, notifications, file change logs, bandwidth limits, backup schedule, and integrity check schedule. We can also “Enable client-side encryption” which we will use to encrypt the data on the cloud.
    Hyper Backup Backup Settings
  14. After proceeding, we will now configure backup rotation settings. I enabled backup rotation with “Smart Recycle”.
    Hyper Backup Rotation Settings
  15. If you enabled encryption, you will be prompted to download your encryption key. Save this file in a safe location as you will need it in the event your NAS fails and you need to access your cloud storage.
    Hyper Backup Synology C2 Encryption PEM key file

And that’s it! You have now configured your NAS to backup to Synology C2 Cloud!

Hyper Backup NAS Backup to Synology C2 Cloud
Hyper Backup NAS Backup to Synology C2 Cloud

We have now configured the backup job. Let’s go ahead and kick off a backup by clicking on “Back up now”.

And that’s it! After some time, your backup should complete and you data should be safe on the Synology C2 Cloud.

What happens when your run of out space

This is a question many of you will be asking, and I was curious myself. I went ahead and uploaded a bunch of garbage data to max out my plan.

Synology Hyper Backup C2 Storage - Not enough quota available on the target
Synology Hyper Backup C2 Storage – Not enough quota available on the target

To my surprise Synology allowed me to exceed my plan by a decent amount of storage until the backup job went in to a “Suspended” state reporting “Not enough quota available on the target”. I actually exceeded the storage by 128GB!!! This is extremely generous!

Synology C2 Storage - Storage overage
Synology C2 Storage – Storage overage

To find out how well the NAS would recover from this situation, I logged on to the Synology C2 Storage service and increased my purchased storage. I was able to simply click on “Action” and “Resume” the backup without any issues… It actually resumed where it left off.

Conclusion

The Synology C2 Storage Cloud service is a great option for backing up your DiskStation NAS and all your important files to the cloud. In my case I use my NAS as a backup, and then further backup my NAS both to a removable hard drive, and the Synology C2 Cloud, keeping me compliant with the 3-2-1 backup rule.

The pricing is amazing and there’s extra cost to backup and restore, which means no upload or download costs. They even provide de-duplication so you’re not paying to store duplicated data.

As part of the backup process, Hyper Backup only copies over blocks of data and files that have changed, which significantly helps bandwidth usage as only what’s changed is copied.

I highly recommend using Synology C2 Storage with your Synology DiskStation NAS!