May 052018
 
Sophos XG Firewall Azure

When deploying the Sophos XG Firewall on to Microsoft Azure, it seems as if you always need to create a new resource group and are limited to certain regions. This is not the case.

This is true when using the “Sophos XG Firewall on Microsoft Azure” quick start guide here, and/or when using the Microsoft Azure Marketplace. If you attempt to deploy to an existing Resource Group, it will prevent you from doing so. It may also restrict you from using some configurations and virtual machine sizes, and also limit which region you can deploy to.

If you’re like me and already have Infrastructure in Microsoft’s Cloud and want more control over configuration and deployment, then your’re probably looking at deploying in to an existing Resource Group, using a more customized configuration, and may have the requirement to deploy to other Azure regions. See below for my own “Quick start to customized Sophos XG deployments on Microsoft Azure” guide.

Also, another interesting note is that the Sophos XG firewalls on Azure do not support the Microsoft Azure backup features. Since you cannot use the Azure backup features on Azure for the Sophos XG firewalls, you’ll need to use the internal Sophos XG configuration backup function. Once configured, you’ll need to be aware of how to deploy a new XG instance (in to an existing Resource Group) in the event of a failure. If a catastrophic failure does occur, you’ll need to re-provision a new Sophos XG instance on Azure, and then restore the Sophos XG configuration to the VM.

Sophos has alternative methods for deploying to Azure, however they are difficult to find. I actually had to spend quite a bit of time to find information, and then spent tons of time learning how to deploy it properly as well. Sophos has GitHub configured as a repo for their Azure XG deployment templates, along with some other goodies that may come in handy. The URL for their GitHub site is https://github.com/sophos-iaas/xg-azure

Before we begin, we make the following assumptions:

  • You are knowledgeable and skilled with Microsoft Azure
  • You are familiar with deploying VMs and objects to Microsoft Azure
  • You have already configured a resource group and/or have existing objects deployed
  • You have already configured subnets for your Sophos XG (Internal and WAN facing)

How to deploy Sophos XG Firewall on Microsoft Azure to existing Resource Group in any region.

Now let’s get started. Here’s my Quick start guide to customized Sophos XG deployments on Microsoft Azure (in to existing Resource Groups).

  1. Go to https://github.com/sophos-iaas/xg-azure and read the page. Once you’re ready to start, scroll down to “Deployment via template”, and select the button that says “Deploy to Azure”.
    Sophos XG GitHub Site
  2. Log in to Microsoft Azure (if you’re not already) and you’ll see the template ready to be deployed. Before filling out the fields, I would recommend looking at what fields are required, and documenting which sections you need to fill out with what details you will populate. I always record and document these settings so that if an issue ever pops up, or if I need to re deploy, I’ll have the previous settings immediately available. If you’re deploying in to an existing Resource Group, configure the fields to reflect it. To proceed with the template deployment, populate the fields, review the ToS, and continue if you accept the terms. See below for the blank template.
    Sophos XG Azure Deployment Template Page 1
    Sophos XG Azure Deployment Template Page 2
  3. After populating the template and executing the deployment, if all goes well you’re Sophos XG firewall should now be deployed in to your existing Resource Group. Please see below for an example of a successful deployment.
    Sophos XG Azure Successful Deployment Template Page 1
    Sophos XG Azure Successful Deployment Template Page 2
  4. And that is it! You can now head over to “All Resources” to view all the objects and make sure everything is configured properly. After double or triple checking your config, and checking/configuring your security groups, you can power up the VM and start playing with your new Sophos XG Firewall deployed on Microsoft Azure.

Final Thoughts

After doing all this you should be able to deploy these on the fly whenever needed. As mentioned above, this is critical knowledge for Sophos XG admins using Microsoft Azure as this process needs to be known for backup and disaster recovery purposes. Also, chances are you’ll need to attempt deployment multiple times before successfully deploying the instance. I had 4 or 5 unsuccessful deployment attempts due to various issues such as mis-typed fields, blank fields, and incorrectly populated fields, so don’t feel bad if it takes you some time. It’s a learning process!

Leave a comment and let me know how your deployment went and if you have any questions!

May 012018
 
Fedora Logo

When attempting to upgrade from Fedora Core 27 to Fedora Core 28, it may fail on the nss-pem package.

I spent some time trying to find the solution for this, and came across numerous posts on the “Red Hat Bugzilla”, particularly this post.

Unfortunately no fix was found.

See below for an example on the failed upgrade output:

[root@SYSTEMZ01 ~]# dnf system-upgrade download --releasever=28
Before you continue ensure that your system is fully upgraded by running "dnf --refresh upgrade". Do you want to continue [y/N]: y
Fedora 28 - x86_64 - Updates
$
Fedora 28 - x86_64
$
google-chrome
$
RPM Fusion for Fedora 28 - Free - Updates
$
RPM Fusion for Fedora 28 - Free
$
RPM Fusion for Fedora 28 - Nonfree - Updates
$
RPM Fusion for Fedora 28 - Nonfree
$
skype (stable)
$
Last metadata expiration check: 0:00:00 ago on Tue 01 May 2018 04:28:04 PM MDT.
Error:
 Problem: nss-pem-1.0.3-6.fc27.i686 has inferior architecture
  - nss-pem-1.0.3-6.fc27.x86_64 does not belong to a distupgrade repository
  - problem with installed package nss-pem-1.0.3-6.fc27.i686

To resolve this, manually install the nss-pem packages from FC28 prior to the upgrade using the following command.

dnf install nss-pem-1.0.3-9.fc28 --releasever=28

After doing so, re-attempt to upgrade and the upgrade should now proceed.

Apr 292018
 
Directory Services Restore Mode

Running Veeam Backup and Replication, a Microsoft Windows Server Domain Controller may boot in to safe mode and directory services restore mode.

About a week ago, I loaded up Veeam Backup and Replication in to my test environment. It’s a fantastic product, and it’s working great, however today I had a little bit of an issue with a DC running Windows Server 2016 Server Core.

I woke up to a notification that the backup failed due to a VSS snapshot issue. Now I know that VSS can be a little picky at times, so I decided to restart the guest VM. Upon restarting, she came back up, was pingable, and appeared to be running fine, however the backup kept failing with new errors, the event log was looking very strange on the server, and numerous services that were set to automatic were not starting up.

This specific server was installed using Server Core mode, so it has no GUI and is administered via command prompt over RDP, or via remote management utilities. Once RDP’ing in to the server, I noticed the “Safe Mode” branding on each corner of the display, this was very odd. I restarted the server again, this time manually trying to start Active Directory Services manually via services.msc.

This presented:

Event ID: 16652
Source: Directory-Services-SAM
General Description: The domain controller is booting to directory services restore mode.

Screenshot:

Directory Services Restore Mode

The domain controller is booting to directory services restore mode.

 

This surprised me (and scared me for that matter). I immediately started searching the internet to find out what would have caused this…

To my relief, I read numerous sites that advise that when an active backup is running on a guest VM which is a domain controller, Veeam activates directory services restore mode temporarily, so in the event of a restore, it will boot to this mode automatically. In my case, the switch was not changed back during the backup failure.

Running the following command in a command prompt, verifies that the safeboot switch is set to dsrepair enabled:

bcdedit /v

To disable directory services restore mode, type the following in a command prompt:

bcdedit /deletevalue safeboot

Restart the server and the issue should be resolved!

Apr 242018
 
DNS

At 5:00AM MST (April 24th, 2018) this morning, I noticed DNS (Domain Name Service) name resolution is failing on numerous internet domains. After further troubleshooting, I realized it’s the root servers that provide DNS that are failing to resolve these hosts.

Some users on providers that cache records may not immediately notice the issues as their ISP has cached records. Doing a search on Twitter confirms numerous people are reporting issues with DNS across numerous providers and ISPs, Google DNS Servers, and Amazon DNS Servers.

 

Update 6:22AM MST:

On my DNS server, I’m noticing the problematic domains that aren’t providing DNS records, are loading NS records that use awsdns-XX dns servers. This could show a problem with Amazon AWS DNS servers. I will continue to try to identify where the issues are.

Update 6:32AM MST:

I’ve noticed that Amazon AWS has since added a “Recent Event” on their service status page for “Amazon Route 53”: “5:19 AM PDT We are investigating reports of problems resolving some DNS records hosted on Route53 using the third party DNS resolvers 8.8.8.8 and 8.8.4.4 . DNS resolution using other third-party DNS resolvers or DNS resolution from within EC2 instances using the default EC2 resolvers are not affected at this time.”

Update 6:45AM MST:

It appears that there are issues with Amazon’s Route 53 DNS service which provides cloud based DNS services. Trying to view https://aws.amazon.com/route53/ results in page load errors.

Update 7:06AM MST:

Another update from Amazon on their service status page for “Amazon Route 53”: 5:49 AM PDT We have identified the cause for an elevation in DNS resolution errors using third party DNS resolvers 8.8.8.8 / 8.8.4.4 and are working towards resolution. DNS resolution using other third-party DNS resolvers or DNS resolution from within EC2 instances using the default EC2 resolvers continues to work normally.

Update 3:50PM MST:

It appears this was actually a malicious attack including DOS, a man in the middle attack, and an attempt to compromise users accounts. Information can be found at https://doublepulsar.com/hijack-of-amazons-internet-domain-service-used-to-reroute-web-traffic-for-two-hours-unnoticed-3a6f0dda6a6f (thanks Juk for this post), and https://www.bleepingcomputer.com/news/security/hacker-hijacks-dns-server-of-myetherwallet-to-steal-160-000/ .

Apr 232018
 

Ready to jump the gun and upgrade to vSphere 6.7? Hold on a moment…

You’ll remember some time ago VMware announced they are dropping support for vSphere vDP (vSphere Data Protection). If you’re running this in your environment, it will break when upgrading to vSphere 6.7.

A better idea would be to migrate over to a product like Veeam, however please note that as of this date, Veeam does not officially support vSphere 6.7. Support should be coming in the next major update.