Sep 302021
 
ISRG and Let's Encrypt

Today, the DST Root CA X3 certificate expired, leaving many devices on the internet having issues connecting to services and certificates that use this Root CA, including those using Let’s Encrypt certificates.

Some of these problematic devices include Samsung Galaxy phones, iPhones, VDI zero and thin clients, and even Sophos UTM firewalls.

In my environment, I noticed a number of issues when browsing to websites that use the free Let’s Encrypt certificates, as the Web Protection Web Filtering service on my Sophos UTM firewall would report the certificate has expired and not allow me access to the websites using it.

The Problem

Let’s Encrypt originally used the “DST Root CA X3” certificate to issue Let’s Encrypt certificates. However, as time has passed and the service has been used more, they now use “ISRG Root X1” and “ISRG Root X2” as Root CA’s and “Let’s Encrypt R3” as an intermediate certificate.

Older devices may be using the older Root CA which expired today (September 30th, 2021). Please see https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ for more information.

The Fix

To fix this issue, you need to add the 2 new Root CAs to your computer or device.

Root CA Certificates (PEM format):

Intermediate Certificate (PEM format):

You can download them by clicking the links above or go to https://letsencrypt.org/certificates/ for more information and to download if you don’t trust the above links.

After downloading and adding these Root CAs and the Intermediate CA to your computer or device, you should have the full certificate chain to validate the Let’s Encrypt certificates. You only need to add the two root certificates. The Let’s Encrypt certificates that are used on websites that you visit and that you might have deployed on your servers should now work without any issues.

If you’re still having issues, you can try deleting the “DST Root CA X3” certificate from your existing Root CAs. Also, you may need to close and reopen any software and/or browsers for it to work with the new certificate.

HTTPS Scanning/Filtering Firewall Fix (Sophos UTM as example)

If you have a firewall that scans HTTPs traffic, you’ll need to add the two root certificates above to the HTTPS Certification authority list.

As an example, to fix this on the Sophos UTM firewall, follow the instructions below:

  1. Download the 3 certificates above.
  2. Log on to your Sophos UTM
  3. Navigate to “Web Protection”, “Filtering Options”, and “HTTPS CAs” tab.
  4. Disable the old “Digital Signature Trust Co. DST Root CA X3” Certificate in the list.
  5. Using the “Upload local CA”, browse to and select 1 of the 3 certificates, then click upload.
  6. Repeat step 5 for each of the 3 certificates listed above.
  7. The issue has been fixed! You should now see all 3 certificates in the “Local verification CAs” list.

The steps should be similar for other firewalls that provide HTTPS Scanning and Filtering.

Jun 012020
 

If you’re running a Sophos UTM firewall, you may start noticing websites not loading properly, or presenting an error reporting that a root CA has expired.

PLEASE NOTE: If you are experiencing this on September 2021 or later, please see DST Root CA X3 Certificate Expiration Problems and Fix.

The error presented is below:

Sectigo COMODO CA Cetificate Untrusted Website Certificate has expired

Webpages that do not present an error may fail to load, or only load partial parts of the page.

Update June 3rd 2020 – There are reports that this issue is also occurring with other vendors security solutions as well (such as Palo Alto Firewalls).

The Issue

This is due to some root CA (Certificate Authority) certificates expiring.

Particularly this involves the following Root CA Certificates:

  • AddTrust AB – AddTrust External CA Root
  • The USERTRUST Network – USERTrust RSA Certification Authority
  • The USERTRUST Network – USERTrust ECC Certification Authority

Read more about the particular issue here: https://support.sectigo.com/Com_KnowledgeDetailPage?Id=kA03l00000117LT

The Fix

To resolve this, we must first disable 3 of the factory shipped Root CA’s (listed above) on the Sophos UTM, and then upload the new Root CAs.

You’ll need to go to the following links (as referenced on Sectigo’s page above, and download the new Root CAs:

USERTrust RSA Root CA (Updated) – https://crt.sh/?id=1199354

USERTrust ECC Root CA (Updated) – https://crt.sh/?id=2841410

When you go to each of the above pages, click on “Certificate” as shown below, to download the Root CA cert.

Download Updated Root CAs Example

Do this for both certificates and save to your system.

Now we must update and fix the Sophos UTM:

  1. Log on to your Sophos UTM Web Interface
  2. Navigate to “Web Protection”, then “Filtering Options”, then select the “HTTPS CAs” tab.
  3. Browse through the list of “Global Verification CAs” and disable the following certificates:
    1. AddTrust AB – AddTrust External CA Root
    2. The USERTRUST Network – USERTrust RSA Certification Authority
    3. The USERTRUST Network – USERTrust ECC Certification Authority
  4. Scroll up and under “Local Verification CAs”, use the “Upload local CA” to upload the 2 new certificates you just downloaded.
  5. Make sure they are enabled.

After you complete these steps, verify they are in the list.

New USERTrust Root CAs Enabled

After performing these steps you must restart the HTTPS Web filter Scanning services or restart your Sophos UTM.

The issue should now be resolved. Leave a comment and let me know if it worked for you!

Aug 092019
 
Sophos UTM with SFP Modules Picture

Today (August 9th, 2019, starting in the early morning) I noticed that numerous Sophos UTM firewalls were sending the notification “The spam filter daemon is unable to reach the database servers via HTTP. Please make sure that the device is able to send HTTP (TCP port 80) requests to the Internet. You may have to allow such traffic on upstream devices.”.

Everything is still functioning and upon troubleshooting I noticed that nothing had been changed, nor was broken. I believe this is a service outage of some type.

This issue has also been reported by numerous other users here: https://community.sophos.com/products/unified-threat-management/f/mail-protection-smtp-pop3-antispam-and-antivirus/114516/getting-spam-filter-cannot-query/411525

I will be updating this post with more information, leave a comment if you have information, or if the issue is also happening to you!

May 062018
 
Sophos XG and Sophos UTM

Today I’m going to be talking about connecting a Sophos XG firewall to a Sophos UTM firewall for a site-to-site VPN connection specifically using SSL tunneling. Furthermore we are doing this to connect a Microsoft Azure Virtual Network (using a Sophos XG instance) to an On-Premise LAN (running a Sophos UTM).

This type of connection and configuration is standard for corporations, businesses, and organizations who have workloads on Microsoft Azure and need to connect their Azure environment to their corporate LAN. To learn how to deploy Sophos XG in Microsoft Azure, please read this post.

WARNING AND DISCLAIMER: Following the steps in this document if done incorrectly or if your environment is different from the one used in this example, can cause network connectivity issues or a loss of connectivity. An assumption is made that you are skillful enough to know what tasks you are performing and what result they may have on your own environment. You may need to revert these steps if connectivity is lost to restore access to your environment.

Let’s ask some key questions and get some answers:

  • Why are we using both (2) products, Sophos XG and Sophos UTM for the connection instead of using the same product on each end?
    • Sophos only supports deployment of the Sophos XG on Microsoft Azure. Sophos UTM cannot be deployed on Azure. Numerous companies and organizations are still using the Sophos UTM product for their on-premise IT infrastructure. There is a need to have these different products co-exist and function together, like in this specific example.
  • Why are we using a Sophos XG Appliance/Instance to handle the VPN on Microsoft Azure, instead of using the Microsoft branded Azure VPN Gateway?
    • Microsoft Azure has a VPN gateway appliance which can handle the Azure side of the VPN connection, however this is a resource that costs money (instance time and instance data transfer) and has limited configuration options. Numerous companies and organizations are using Sophos XG instances on Azure to protect their internet facing workloads already. Instead of paying for 2 resources (Sophos XG and Microsoft VPN Gateway), you can consolidate and use one for both. You also have extra functionality and security options when using the Sophos XG instance to handle VPN traffic (IPS, strict firewall rules, packet inspection, etc). The Sophos firewalls can be centrally managed/monitored via Sophos Management Products (Sophos SUM (Sophos UTM Manager), and/or Sophos CFM (Sophos Central Firewall Manager)).
  • Why are we using an SSL VPN connection instead of IPSec or other type of VPN?
    • Microsoft Azure blocks some IP Protocol traffic within Virtual Networks. As quoted: “IP-in-IP encapsulated packets, and Generic Routing Encapsulation (GRE) packets are blocked within VNets” (per https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-faq), which means that while you could configure an IPsec tunnel, none of the traffic would pass through the Virtual Networks. We utilize SSL VPNs to overcome this limitation as all traffic goes over the SSL connection.

Let’s address what components we will discuss in this post:

The information inside of this post can be used for any of the 4 components above and don’t necessarily have to be used in the same configuration. An example: This guide could be used by someone wanting to connect an on-premise XG and on-premise UTM unit together via SSL VPN (with no need or use for Azure). Another example: The section on routing tables can provide information for someone using a different network security product on Microsoft Azure. However, the ultimate goal of this article is to address all four of the components together for a complete end to end deployment.

Now let’s get to the configuration of each of the four components.

Deploy a Sophos XG instance/appliance to Microsoft Azure

In a previous post, I covered how to deploy a new Sophos XG firewall appliance/instance to Microsoft Azure, specifially allowing deployment to existing resource groups. The full URL (and instructions) can be found at https://www.stephenwagner.com/2018/05/05/deploy-sophos-xg-firewall-microsoft-azure-existing-resource-group/

Sophos XG to Sophos UTM SSL VPN Connection Configuration and Encryption Settings

We will configure the SSL VPN settings on both the Microsoft Azure Sophos XG appliance/instance, and the on-premise Sophos UTM appliance/instance. Afterwards, we will create a tunnel, configure it, enable it, and establish connectivity between the two Sophos firewall instances. During this process, we’ll be configring the SSL VPN settings on both appliance/instances, configuring the tunnel, configuring encryption settings, and establishing VPN communication.

Please note, with this configuration there are essentially 3 networks: the Azure network, the SSL transit network (this is the network in between the networks we are connecting that is part of the SSL VPN), and the on-premise network. When you configure your firewall rules (not in the scope of this document), you must factor this in and allow applicable traffic to/from the SSL network so that the packets can pass. This SSL transit network is specified in the “Show VPN Settings” on the XG unit under “SSL VPN”, and “IPv4 Lease Range”. This network must be different and not overlap any subnets you are using on both your Azure network, and on-premise network. In my case I chose something way off in an entirely different IP space (172.16.0.0), and I suggest you do as well.

Follow these steps to configure the SSL VPN settings on both Sophos XG and UTM appliances/instances.

  1. Log on to the Sophos XG appliance, select “VPN” under “Configure” on the left hand side, and then select the gear icon with “Show VPN Settings” on the top right. See example.

    Sophos XG Show VPN Settings

    Sophos XG Show VPN Settings

  2. Configure your “SSL VPN Settings” and “Cryptographic Settings” to be similar to the example below. Please modify to reflect your own environment. Cryptographic settings should match example below. Please note that the “IPv4 Lease Range” is not for your Azure or Internal LAN subnet, but actually the subnet used by the SSL VPN server. This value has to be an entirely new subnet dedicated for SSL VPN functions.

    Sophos XG SSL VPN Settings

    Sophos XG SSL VPN Settings

  3. Log on to the Sophos UTM appliance, select “Site-to-Site VPN”, then select “SSL”, then click on the “Advanced” tab.
  4. Configure your “Advanced” settings to reflect the following below.

    Sophos UTM SSL Site-to-Site VPN Advanced Settings

    Sophos UTM SSL Site-to-Site VPN Advanced Settings

You have now configured the general SSL VPN Advanced settings, we can now move on to configuring the tunnel itself.

To configure the SSL Site to Site VPN tunnel between the Sophos appliances, we’ll need to configure the Sophos XG (on Azure) to act as a server, and the Sophos UTM (on prem) which will act as the client. Side note: In my own testing, I found that the XG had to be the server in order to get them to connect.

To configure the SSL VPN tunnel Server on the Sophos XG:

  1. Log on to your Sophos XG interface, click on “VPN” under “Configure” on the left hand side, and then choose “SSL VPN (Site-to-Site)” from the top. Then click “Add” under the “Server” section. As shown below.

    Sophos XG SSL VPN (Site-to-Site) Settings

    Sophos XG SSL VPN (Site-to-Site) Settings

  2. Now give the VPN connection a name, enter a friendly description, and specify Local Networks (Azure Subnets), and Remote Networks (On-Prem Networks). And then click “Save” as shown below.

    Sophos XG Create new SSL VPN Server

    Sophos XG Create new SSL VPN Server

  3. Now we need to download the configuration so that we can load it on to your Sophos UTM so it know’s how to connect. While still in the “SSL VPN (Site-to-Site)” window, look for the column called “Manage”, and click the download icon (you can confirm it’s download via the mouse-over description). As shown below.

    Sophos XG SSL VPN Server Download Config

    Sophos XG SSL VPN Server Download Config

  4. Save this file to your computer as we’ll need it for configuring the Sophos UTM

To configure the SSL VPN tunnel Client on the Sophos UTM:

  1. Log on to your Sophos UTM web interface, click on “Site-to-Site VPN” on the left hand side, and then select “New SSL Connection”. As shown below in step 2.
  2. Set the Connection type to “Client”, give it a friendly name. Now click on the folder next to “Configuration File” to brows to the VPN config file (this is the file we downloaded above from the Sophos XG unit). In my case, I also selected the “Override peer hostname” as I wanted to over the hostname of the Sophos XG unit (to avoid problems I chose an IP address). This value sets the hostname of the server that the UTM is connecting to. We also uncheck the “Automatic firewall rules” as we don’t want any rules automatically created, we want to specify only what is needed.

    Sophos UTM Create New SSL VPN Connection

    Sophos UTM Create New SSL VPN Connection

  3. Hit Save, and you should be left with something like below.

You have now fully configured the SSL Site-to-Site VPN tunnel between your Sophos XG and Sophos UTM instances.

To confirm a functioning VPN tunnel on your Sophos XG unit, you should see something similar to below.

Sophos XG SSL VPN (Site-to-Site) Active VPN Status

Sophos XG SSL VPN (Site-to-Site) Active VPN Status

To confirm a functioning VPN tunnel on your Sophos UTM unit, you should see something simialar to below.

Sophos UTM Active VPN Status

Sophos UTM Active VPN Status

Please note that a bug with XG to UTM VPN, is that on the Sophos UTM reports the active subnets as “unknown” as shown above on both sides. This can be safely ignored.

You can start to test some basic communication, however you still need to create firewall rules. Please note that the Azure network will not be routable until you continue the steps below and configure proper routing.

Microsoft Azure VNet (Virtual Network) custom routing table

When you create a Virtual Network (VNet) in Microsoft Azure, Azure will handle the routing for you automatically. It will create routers and other “instances” to handle network connectivity as you provision new subnets, gateways, devices, and network paths. Since we are deploying our own router, we want to override these default routing tables that are automatically created.

When looking at our target configuration, our Sophos XG unit will have an internet facing static IP, and will be handling communication between our internal network (and hosts), the internet (outside world), and our internal on-premise network (LAN). Because of this, we have to make changes to our Azure enviroinment so that the default subnet network route becomes the Internal IP Address of the Sophos XG firewall. We need to configure routes for both our Azure subnets (if wanted), our corporate on-premise LAN, and our catch-all route for internet access (0.0.0.0/0).

Once we create our own routing table, we’ll need to assign it to specific subnets to make those specific subnets enforce the routes.

To create a custom Route Table:

  1. Log in to the Azure Portal, View “All Resources”, click on the “Add” button at the top to create a new resource, or simply click on “Create a resource” on the top left of the Azure Portal.
  2. Select Networking on the left side of the table, then select “Route Table”. See example below.

    Sophos XG Azure Add Resource Route Table

    Sophos XG Azure Add Resource Route Table

  3. Populate the fields, select the subscription, resource group, and the Location of where you’d like to place this. For “Resource Group”, select “Use existing”, and then specify the Resource Group you are attaching this to.

    Sophos XG Azure Create Route Table

    Sophos XG Azure Create Route Table

  4. Select “Create” to complete the creation of the route table.
  5. Now that we have a customer routing table, we want to create a route. With the “Route Table” object still open, open the “Routes” tab to open the page from the column on the left.
  6. Click on Add, and then create a route for the CIDR block that covers both your Azure subnets, and corporate on-premise LAN subnet. PLEASE NOTE: This CIDR block (Address prefix) should be large enough that it includes both your Azure subnets and your on-premise network. This will make this rule apply to all traffic destined for those networks. Under “Next Hop Type”, select “Virtual Appliance”, and then enter the IP address of your Sophos XG LAN Network interface. Then click save.

    Sophos XG Azure Route Table LAN Route

    Sophos XG Azure Route Table LAN Route

  7. Click on Add, and then create a catch all route for the internet/WAN with an address prefix of 0.0.0.0/0. Under “Next Hop Type”, select “Virtual Appliance”, and then enter the IP address of your Sophos XG LAN Network interface. Then click save.

    Sophos XG Azure Route Table Internet Route

    Sophos XG Azure Route Table Internet Route

  8. The Route’s section should look similar to this.

    Sophos XG Azure Route Table Route List

    Sophos XG Azure Route Table Route List

  9. Now we need to assign the route table to the chosen subnets. This will apply the routes we created above to specific subnets on our Azure Virtual Network (VNet). With the “Route Table” object still open, open the “Subnets” page from the column on the left.
  10. Click on “Associate” to add this to an existing subnet. And then associate it to your Azure Virtual Network subnet where the XG and your VMs reside.

    Sophos XG Azure Route Table Associate Subnet

    Sophos XG Azure Route Table Associate Subnet

  11. Now completed, the “Overview” tab should look similar to this.

    Sophos XG Azure Route Table Overview

    Sophos XG Azure Route Table Overview

You’ve successful configured a custom routing table for your Microsoft Azure subnet which will route packets destined for other subnets (including internet/WAN) to your Sophos XG for further routing.

Microsoft Azure enable IP Forwarding on Virtual Network Interface

In order for a VM (Virtual Machine) to have the ability to forward and route packets, we need to enable “IP Forwarding” on both the Internal and External NIC of the Sophos XG appliance running on Azure.

To enable this:

  1. Log in to the Azure Portal, view “All Resources” and select one of the “Network interface” objects for your Sophos XG appliance/instance.
  2. Click on “IP configurations” under the “SETTINGS” group.
  3. Look for “IP forwarding” under “IP forwarding settings”. Set this toggle to “Enabled” as per below.

    Enable IP Forwarding on Microsoft Azure Network Interface

    Enable IP Forwarding on Microsoft Azure Network Interface

  4. Click Save
  5. Repeat for the other Sophos Network interface (both External and Internal need this enabled)

IP Forwarding has now been enabled. The Sophos XG appliance can now successfully route packets in your Microsoft Azure Virtual Network (VNet).

Conclusion

At this point you’ll have everything configured. You’ll have a SSL VPN between your Azure hosted Sophos XG instance and your on-premise Sophos UTM, as well as connectvity between both of the networks. You will now need to configure all your firewall rules to only permit the traffic you want to traverse from internal-azure to internal-onprem as well as external WAN traffic (this is beyond the scope of this document). You need to take care in making sure you only permit traffic that should be going over these network links. Now that both networks are connected, it provides another means to connect and communicate with the other networks which increases your security risks. You’re not only securing against one internet connection on one LAN, but 2 internet connections across 2 networks.

In my scenario by configuring this I was able to decommission the Microsoft Azure VPN Gateway (minimizing costs), and have my own security appliance/instance handle the communication between both networks and also protect both networks with all the fancy features that the Sophos XG and UTM line offer.

Leave a comment with feedback!

May 052018
 
Sophos XG Firewall Azure

When deploying the Sophos XG Firewall on to Microsoft Azure, it seems as if you always need to create a new resource group and are limited to certain regions. This is not the case.

This is true when using the “Sophos XG Firewall on Microsoft Azure” quick start guide here, and/or when using the Microsoft Azure Marketplace. If you attempt to deploy to an existing Resource Group, it will prevent you from doing so. It may also restrict you from using some configurations and virtual machine sizes, and also limit which region you can deploy to.

If you’re like me and already have Infrastructure in Microsoft’s Cloud and want more control over configuration and deployment, then your’re probably looking at deploying in to an existing Resource Group, using a more customized configuration, and may have the requirement to deploy to other Azure regions. See below for my own “Quick start to customized Sophos XG deployments on Microsoft Azure” guide.

Also, another interesting note is that the Sophos XG firewalls on Azure do not support the Microsoft Azure backup features. Since you cannot use the Azure backup features on Azure for the Sophos XG firewalls, you’ll need to use the internal Sophos XG configuration backup function. Once configured, you’ll need to be aware of how to deploy a new XG instance (in to an existing Resource Group) in the event of a failure. If a catastrophic failure does occur, you’ll need to re-provision a new Sophos XG instance on Azure, and then restore the Sophos XG configuration to the VM.

Sophos has alternative methods for deploying to Azure, however they are difficult to find. I actually had to spend quite a bit of time to find information, and then spent tons of time learning how to deploy it properly as well. Sophos has GitHub configured as a repo for their Azure XG deployment templates, along with some other goodies that may come in handy. The URL for their GitHub site is https://github.com/sophos-iaas/xg-azure

Before we begin, we make the following assumptions:

  • You are knowledgeable and skilled with Microsoft Azure
  • You are familiar with deploying VMs and objects to Microsoft Azure
  • You have already configured a resource group and/or have existing objects deployed
  • You have already configured subnets for your Sophos XG (Internal and WAN facing)

How to deploy Sophos XG Firewall on Microsoft Azure to existing Resource Group in any region.

Now let’s get started. Here’s my Quick start guide to customized Sophos XG deployments on Microsoft Azure (in to existing Resource Groups).

  1. Go to https://github.com/sophos-iaas/xg-azure and read the page. Once you’re ready to start, scroll down to “Deployment via template”, and select the button that says “Deploy to Azure”.
    Sophos XG GitHub Site
  2. Log in to Microsoft Azure (if you’re not already) and you’ll see the template ready to be deployed. Before filling out the fields, I would recommend looking at what fields are required, and documenting which sections you need to fill out with what details you will populate. I always record and document these settings so that if an issue ever pops up, or if I need to re deploy, I’ll have the previous settings immediately available. If you’re deploying in to an existing Resource Group, configure the fields to reflect it. To proceed with the template deployment, populate the fields, review the ToS, and continue if you accept the terms. See below for the blank template.
    Sophos XG Azure Deployment Template Page 1
    Sophos XG Azure Deployment Template Page 2
  3. After populating the template and executing the deployment, if all goes well you’re Sophos XG firewall should now be deployed in to your existing Resource Group. Please see below for an example of a successful deployment.
    Sophos XG Azure Successful Deployment Template Page 1
    Sophos XG Azure Successful Deployment Template Page 2
  4. And that is it! You can now head over to “All Resources” to view all the objects and make sure everything is configured properly. After double or triple checking your config, and checking/configuring your security groups, you can power up the VM and start playing with your new Sophos XG Firewall deployed on Microsoft Azure.

Final Thoughts

After doing all this you should be able to deploy these on the fly whenever needed. As mentioned above, this is critical knowledge for Sophos XG admins using Microsoft Azure as this process needs to be known for backup and disaster recovery purposes. Also, chances are you’ll need to attempt deployment multiple times before successfully deploying the instance. I had 4 or 5 unsuccessful deployment attempts due to various issues such as mis-typed fields, blank fields, and incorrectly populated fields, so don’t feel bad if it takes you some time. It’s a learning process!

Leave a comment and let me know how your deployment went and if you have any questions!

Jan 182018
 

The Problem

I run a Sophos UTM firewall appliance in my VMware vSphere environment and noticed the other day that I was getting warnings on the space used on the ESXi host for the thin-provisioned vmdk file for the guest VM. I thought “Hey, this is weird”, so I enabled SSH and logged in to check my volumes. Everything looked fine and my disk usage was great! So what gives?

After spending some more time troubleshooting and not finding much, I thought to myself “What if it’s not unmapping unused blocks from the vmdk to the host ESXi machine?”. What is unampping you ask? When files get deleted in a guest VM, the free blocks aren’t automatically “unmapped” and released back to the host hypervisor in some cases.

Two things need to happen:

  1. The guest VM has to release these blocks (notify the hypervisor that it’s not using them, making the vmdk smaller)
  2. The host has to reclaim these and issue the unmap command to the storage (freeing up the space on the SAN/storage itself)

On a side note: In ESXi 6.5 and when using VMFS version 6 (VMFS6), the datastores can be configured for automatic unmapping. You can still kick it off manually, but many administrators would prefer it to happen automatically in the background with low priority (low I/O).

Most of my guest VMs automatically do the first step (releasing the blocks back to the host). On Windows this occurs with the defrag utility which issues trim commands and “trims” the volumes. On linux this occurs with the fstrim command. All my guest VMs do this automatically with the exception being the Sophos UTM appliance.

The fix

First, a warning: Enable SSH on the Sophos UTM at your own risk. You need to know what you are doing, this also may pose a security risk and should be disabled once your are finished. You’ll need to “su” to root once you log in with the “loginuser” account.

This fix not only applies to the Sophos UTM, but most other linux based guest virtual machines.

Now to fix the issue, I used the “df” command which provides a list of the filesystems, their mount points, and storage free for those fileystems. I’ve included an example below (this wasn’t the full list):

hostname:/root # df
Filesystem                       1K-blocks     Used Available Use% Mounted on
/dev/sda6                          5412452  2832960   2281512  56% /
udev                               3059712       72   3059640   1% /dev
tmpfs                              3059712      100   3059612   1% /dev/shm
/dev/sda1                           338875    15755    301104   5% /boot
/dev/sda5                         98447760 13659464  79513880  15% /var/storage
/dev/sda7                        129002700  4624468 117474220   4% /var/log
/dev/sda8                          5284460   274992   4717988   6% /tmp
/dev                               3059712       72   3059640   1% /var/storage/chroot-clientlessvpn/dev


You’ll need to run the fstrim command on every mountpoint for file systems “/dev/sdaX” (X means you’ll be doing this for multiple mountpoints). In the example above, you’ll need to run it on “/”, “/boot”, “/var/storage”, “/var/log”, “/tmp”, and any other mountpoints that use “/dev/sdaX” filesytems.

Two examples:

fstrim / -v

fstrim / -v

 

 

fstrim /var/storage -v

fstrim /var/storage -v

 

 

Again, you’ll repeat this for all mount points for your /dev/sdaX storage (X is replaced with the volume number). The command above only works with mountpoints, and not the actual device mappings.

Time to release the unused blocks to the SAN:

The above completes the first step of releasing the storage back to the host. Now you can either let the automatic unmap occur slowly overtime if you’re using VMFS6, or you can manually kick it off. I decided to manually kick it off using the steps I have listed at: https://www.stephenwagner.com/2017/02/07/vmfs-unmap-command-on-vsphere-6-5-with-vmfs-6-runs-repeatedly/

You’ll need to use esxcli to do this. I simply enabled SSH on my ESXi hosts temporarily.

Please note: Using the unmap command on ESXi hosts is very storage I/O intensive. Do this during maintenance window, or at a time of low I/O as this will perform MAJOR I/O on your hosts…

I issue the command (replace “DATASTORENAME” with the name of your datastore):

esxcli storage vmfs unmap --volume-label=DATASTORENAME --reclaim-unit=8

This could run for hours, possibly days depending on your “reclaim-unit” size (this is the block size of the unit you’re trying to reclaim from the VMFS file-system). In this example I choose 8, but most people do something larger like 100, or 200 to reduce the load and time for the command to complete (lower values look for smaller chunks of free space, so the command takes longer to execute).

I let this run for 2 hours on a 10TB datastore, however it may take way longer (possibly 6+ hours, to days).

Finally, not only are we are left with a smaller vmdk file, but we’ve released the space back to the SAN as well!

Oct 192017
 

In the past few days, I’ve noticed that some Sophos UTM firewalls I manage for clients haven’t been sending their daily reports (or other notification e-mails). When I first noticed this, checking my own SMTP proxy, I noticed that the e-mails were being sent from the firewalls, but were being dropped due to an SPF check failure.

Originally I thought this may have just been an overnight glitch with the DNS providers, however I later noticed that it’s stopped all e-mails coming from all the UTMs.

Further investigation, I realized that by default, the Sophos UTMs send their firewall notifications (and configuration backups) from the domain “fw-notify.net”, specifically, the e-mail address “[email protected]”. That’s when I had a brainfart and realized the e-mails weren’t being sent from my clients owned domains, but this fw-notify.net domain.

It appears that recently some SPF records have been created for the domain “fw-notify.net”, which is what is causing this issue. Also, I’m not quite sure if the domain underwent ownership change, or it his was overlooked by someone at Sophos.

I’m assuming numerous other longtime UTM users will be experiencing this as well.

To fix this, just log in to the problem UTMs, and change the notification Sender address as shown below to a domain you own. I changed mine to [email protected] (which has valid SPF since it’s my domains relay).

Jul 032010
 

I’ve had my main web server directly on the net for some time now. The box runs CentOS and I always have it fully up to date, with a minimal install just to act as a web server.

It’s always concerned me a little bit, the fact is I keep the box up to date as much as possible, but it’s still always in the back of my mind.

This weekend I had some time to mess around with some stuff. I wanted to get it setup behind my Sophos UTM, however I did NOT want it to use the public IP address that it’s setup for as I have numerous static IPs all for different services.

I spent a good 3-4 hours doing lots of searching on Google, and Astaro.org. I saw a few people that wanted to do the same thing as me, but didn’t really find an explanation for anything.

Ultimately I wanted to setup another external IP address on the Sophos UTM software appliance box, and have that external IP dedicated to JUST the web server. Everything else would continue to run as configured before I started modifying anything.

I finally got it going, and I thought I would do a little write up on this since I saw a lot of people were curious, however no one was having luck with it. So far I’ve just done it for my main web server, however in the future I’ll be doing this with a few more external IPs and servers of mine. So let’s log into the Astaro web interface and get started!

PLEASE NOTE: I performed this configuration on Astaro Security Gateway Version 8, this will also work on a Sophos UTM

  1. Configure the additional IP  –              “Interfaces & Routing”, then choose “Interfaces”. Select the “Additional Addresses” tab on the top of the screen. Hit the “New additional address…” button and configure the additional IP. Please note this worked for me as all my static IPs use the same gateway for the most part, if you have multiple statics that use different gateways this may not work for you. In my case I called this address “DA-Web”. Make sure you enable this afterwards by hitting the green light!
  2. Configure the NAT Rules      –              On the left select “Network Security”, then choose the sub item “NAT”. We do not want to touch anything under “Masquerading” so lets go ahead and select the “DNAT/SNAT” tab. In this section we need to create two rules, one for DNAT, and one for SNAT. Keep in mind that “Full NAT” is available, but due to the setup of the traffic initiation I don’t think we want to touch this at all.
    1. Create the DNAT Rule            –              Hit the “New NAT rule” button. Set “Position” to Top”. “Traffic Source” and “Traffic Service” to “Any”. “Traffic Destination” set to the additional address you created (keep in mind this has the same name as the main external, only with the name of the connection inside of it). Set “NAT mode” to “DNAT”. And finally set Destination to the server you want this going to, or create a new definition for the server. Make sure “Automatic packet filter rule” is NOT checked. See image below for my setup.
    2. Create the SNAT Rule            –              Hit the “New NAT rule” button. Set the “Position” to top. “Traffic Source” should be set to the definition you created for the server you are doing this for. “Traffic Service” should be “Any”. “Traffic Destination” should be “Internet”. Keep in mind this is very important, we want to make sure that if you use multiple subnets inside your network that SNAT is ONLY performed when needed when data gets shipped out to the Internet, and NOT when your internal boxes are accessing it. Set “NAT mode” to SNAT. And finally “Source” being the additional IP you created (again this looks like your normal External IP, but hold the mouse over when selecting the definition to make sure it’s the “additional” IP you created). Make sure “Automatic packet filter rule” is NOT checked. See image below for my setup.
    3. Create Packet Filter Rules    –              Now it’s time to open some ports up so that your server can offer services to the internet. This is fairly standard so I’m sure that you can do it on your own. In my example I created a few rules that allowed HTTP, DNS, and FTP from “any” using the service, to the destination “DA-Webserver” to allow the traffic I needed.

This should be it, it should be working now. If you don’t want to create the packet filter rules and want ALL traffic allowed, you can simply forget section c above, and when creating the DNAT and SNAT rules check the “Create automatic packet filter rules” box on both rules. Keep in mind this will be opening your box up to the internet!

If you find this useful, have any questions, or want to comment or tell me how to do it better, please leave me a comment!

Thanks!