Oct 192017
 

In the past few days, I’ve noticed that some Sophos UTM firewalls I manage for clients haven’t been sending their daily reports (or other notification e-mails). When I first noticed this, checking my own SMTP proxy, I noticed that the e-mails were being sent from the firewalls, but were being dropped due to an SPF check failure.

Originally I thought this may have just been an overnight glitch with the DNS providers, however I later noticed that it’s stopped all e-mails coming from all the UTMs.

Further investigation, I realized that by default, the Sophos UTMs send their firewall notifications (and configuration backups) from the domain “fw-notify.net”, specifically, the e-mail address “do-not-reply@fw-notify.net”. That’s when I had a brainfart and realized the e-mails weren’t being sent from my clients owned domains, but this fw-notify.net domain.

It appears that recently some SPF records have been created for the domain “fw-notify.net”, which is what is causing this issue. Also, I’m not quite sure if the domain underwent ownership change, or it his was overlooked by someone at Sophos.

I’m assuming numerous other longtime UTM users will be experiencing this as well.

To fix this, just log in to the problem UTMs, and change the notification Sender address as shown below to a domain you own. I changed mine to fw-notify@mydomainname.com (which has valid SPF since it’s my domains relay).

Oct 182017
 

After installing Windows 10 Fall Creators Update (Windows 10 Version 1709), I’m noticing that on one of my multi-monitor machines it’s showing blue colors as purple on one of the displays.

This is very visible when highlighting text, viewing the blue Facebook logo and banner, or any other blue content. When dragging something across both displays (window is shown on both displays) you can see the color differences. However, one interesting thing, is that when dragging from one display to the other, for the last 10% or so when moving, it’ll quickly change to the proper blue before leaving the display, which means this is software related since it will briefly show the proper blue.

After spending over an hour troubleshooting, it’s totally unrelated to monitor drivers (color configurations), video drivers, etc… and I cannot find any configuration to fix this. Also, searching on the internet I cannot find any other occurrences.

Please comment if you have any information, or are experiencing the same issue!

 

Oct 182017
 

Well, it’s October 18th 2017 and the Fall Creators update (Feature update to Windows 10, version 1709) is now available for download. In my particular environment, I use WSUS to deploy and manage updates.

I went ahead earlier today and approved the updates for deployment, however I noticed an issue on multiple Windows 10 machines, where the Windows Update client would get stuck on Downloading updates 0% status.

I checked a bunch of things, but noticed that it simply couldn’t download the updates from my WSUS server. Further investigation found that the feature updates are packaged in .esd files and IIS may not be able to serve these properly without a minor modification. I remember applying this fix in the past, however I’m assuming it was removed by a prior update on my Windows Server 2012 R2 server.

If you are experiencing this issue, here’s the fix:

  1. On your server running WSUS and IIS, open up the IIS manager.
  2. Expand Sites, and select “WSUS Administration”
  3. On the right side, under IIS, select “MIME Types”
  4. Make sure there is not a MIME type for .esd, if there is, you’re having a different issue, if not, continue with the instructions.
  5. Click on “Add” on the right Actions pane.
  6. File name extension will be “.esd” (without quotations), and MIME type will be “application/octet-stream” (without quotations).
  7. Reset IIS or restart WSUS/IIS server

You’ll notice the clients will not update without a problem! Happy Updating!

Sep 292017
 

There is a new issue starting to be visible in the last couple days that I’ve noticed across 3 fully patched systems (Windows 10 running Outlook 2016 connecting to Exchange 2013).

When using Microsoft Outlook 2016 with Microsoft Exchange 2013, a password prompt becomes visible when opening an attachment in an e-mail. The attachment will open, however the prompt occurs after it’s opened, and only appears if an attachment is opened in the first place. The prompt will not appear if an attachment is never opened or highlighted (selected).

Outlook Password Prompt

When entering AD credentials, the prompt keeps re-appearing. When you hit cancel, Outlook will continue to function. You may also see the prompt shown below.

Exchange Password Prompt

After troubleshooting, I can confirm this is NOT related to any of the traditional “Outlook password prompt” issues that users normally experience due to misconfiguration, and I have a feeling this is related to either an Outlook 2016 update, or an update for Microsoft Windows 10 (and/or Microsoft Windows 7).

I’ve only found one other mention of this occurring on the internet which appeared a day ago, where multiple users are experience the same issue with Microsoft Office 365 with Microsoft Outlook 2016 with multiple operating systems (Windows 10 and Windows 7).

Microsoft Office Version: 1708 (Build 8431.2079)

As of right now I have no information on a fix, but I wanted to post this before other admins start ripping apart their Exchange servers trying to resolve this.

 

Update October 2nd, 2017: I’ve read that someone used the downgrade guide from Microsoft and downgraded their Outlook 2016 client to an earlier “Click-to-Run” 2016 version. This stopped the password prompt so it appears this issue has to do with the latest updates for Microsoft Office (Office 2016 and Office 365).

Jun 012017
 

Today I’m writing about something we all hate, issues with either limited or no cell phone reception. There’s pictures below so please scroll down and check them out!

We’ve all lived in a house or area where there’s no reception at some point in our life. In the house that I’m in right now, I’ve had no or limited reception for the past 2 years. Regularly I have missed calls (phone won’t ring, and I’ll receive a voicemail notification 2 hours later), or people will send me text messages (SMS) and I won’t receive them for hours. Sometimes if someone sends multiple SMS messages, I’ll actually even completely lose reception for 15 minute intervals (phone completely unusable).

This has been extremely frustrating as I use my phone a lot, and while I do have an office line, people tend to call your mobile when they want to get in touch ASAP. It became an even larger problem when clients started texting me for work emergencies. While I always stress to call the office, they are texting these more and more often.

Recently, to make the problem worse I switched from a Microsoft Lumia 950XL to a Samsung Galaxy S8+. When I received my new S8+, my phone wouldn’t even ring at all, while occasionally I could make an outbound call.

 

For these reception issues, there are typically 4 ways to resolve them:

  1. WiFi Calling
    1. Routes calls, SMS/MMS (texting), and cell services through a traditional Wifi access point. Unfortunately Canadian carriers just recently started to implement this, also you’ll need a supported carrier branded phone. Wifi calling usually won’t work if you’re using an unlocked phone, or purchased directly from manufactorer (you’ll need to buy a phone directly from your provider).
    2. Provides easy handoffs from Wifi calling to the native cell towers.
    3. Unfortunately, if you’re in a low reception area, you’re phone will continue to scan and struggle to connect to cell towers (even though it’s sitting in standby). This will consume battery power.
    4. Easy as it requires no special hardware except a phone and carrier that supports the technology.
  2. Femtocell/microcell/picocell
    1. This is a little device that looks similar to your wireless router or wireless access point.
    2. Connects to your provider using your internet connection. The device is essentially a mini cell tower that your phone will connect to using its normal cellular technologies.
    3. These are popular in the United States with multiple carriers providing options, however my provider in Canada doesn’t sell or use these. I could be wrong but I don’t think any providers in Canada carry these.
    4. Easy as it requires only a single small box similar to your wifi router, and a carrier that supports it.
  3. Cell Amplifier / Cell Booster
    1. A device with two antennas, one indoor and one outdoor. Install outdoor antenna facing closest cell tower, install indoor antenna in your house. This boosts and amplifies the signal coming in and going out.
    2. This option is more difficult as it requires mounting an antenna either outdoors (for best reception) or inside of a window. Also cabling must be laid to the booster which must be a specified distance away from the outside antenna. This can be overwhelming and challenging for some.
    3. Most expensive option if you don’t move.
  4. Move to a new house
    1. Most expensive option
    2. Chances it may not correct, or even make your reception issue worse
    3. New neighbors might be crazy

 

In my scenario, I decided to purchase a Wilson Electronics – weBoost Home 4G Cell Phone Booster Kit. With my lack of experience with boosters, I decided to purchase the most cost-effective option that supported LTE and also which was a refurbished unit. I figured if it worked, I could upgrade it in the future to a better model that was brand new and a model higher.

 

Please see the links below for information:

https://wilsonamplifierscanada.ca/ – Canada Online Store

https://www.weboost.ca/ – Manufacturer website with information on products

The model I purchased:

Refurbished Part#: 470101R

https://wilsonamplifierscanada.ca/weboost-home-4g-cell-phone-booster-kit-refurbished-470101r/

New Part#: 470101F

https://wilsonamplifierscanada.ca/weboost-home-4g-cell-phone-signal-booster-470101/

weBoost Home 4G Product Page

https://www.weboost.com/products/connect-home-4g (United States Web Site)

https://www.weboost.ca/products/connect-home-4g (Canada Web Site)

 

Well, after a few weeks the booster finally showed up! Everything was packed nicely, and I was pleasantly surprised about the quality of the materials (antennas, cables) and the unit itself. With my specific unit being a refurbished model, it looked great and you wouldn’t have been able to even notice.

The unit comes with mounting supplies for different mounting options. I could either mount it on a pole (such as the plumbing exhaust port on the roof), against the side of the house, or use the neat window mounting option for window placement (neat little window mount that uses suction cups to affix).

I already was aware of the location of two towers in my area and had previously used cell surveying utilities to find areas where reception was available. If you purchase a cell booster, you can either follow the instructions for finding the best placement with cell service, or you can use apps on your phone to find the best placement.

Here’s some pictures from unboxing and testing. Please click on the image to see a larger version of the image:

weBoost Home 4G 470101

weBoost Home 4G 470101 Cell Booster Kit

 

weBoost Home 4G 470101 Cell Booster

weBoost Home 4G 470101 Cell Booster Unboxed

 

weBoost Home 4G 470101 Cell Booster

weBoost Home 4G 470101 Cell Booster Refurbished

 

weBoost Home 4G 470101 Cell Booster Outside Antenna Window Mount

weBoost Home 4G 470101 Cell Booster Outside Antenna mounted on Window

 

 

weBoost Home 4G 470101 Cell Booster

weBoost Home 4G 470101 Cell Booster

 

weBoost Home 4G 470101 Cell Booster Inside Antenna

weBoost Home 4G 470101 Cell Booster Inside Antenna

 

weBoost Home 4G 470101 Cell Booster Turned on

weBoost Home 4G 470101 Cell Booster Turned on with full Green LED lights (operational)

 

And BAM! That was it, literally on the first test it worked great. Full bars in the basement with my main carrier! I tried a few other locations, and found at an alternative location, my other cell provider (I have 2 phones, with two providers), started to function as well!

 

See below for reception before and after:

 

As you can see there was a vast improvement! I tested it with phone calls, texts, MMS messages, and data, and it all worked fantastic! All lights on the booster were green (orange and/or red lights mean adjustments are needed).

Now since testing was complete, I decided to install it to make it look neat and tidy and hide all the wires.

I decided to leave it using the window mount since it was working so well (this was to avoid having to get on the roof, or drill in to the house). Underneath the window I have a cool-air intake so I was able to fish the antenna wire through the ventilation duct down to the basement. I was able to make everything look neat and tidy.

Below pics are final install:

Installed weBoost Home 4G 470101 Cell Booster

Installed weBoost Home 4G 470101 Cell Booster

 

Installed weBoost Home 4G 470101 Cell Booster

Installed weBoost Home 4G 470101 Cell Booster

 

Installed weBoost Home 4G 470101 Cell Booster Inside Antenna

Installed weBoost Home 4G 470101 Cell Booster Inside Antenna

 

The entire process was extremely easy and I’m very happy with the result. I’d highly recommend this to anyone with reception issues. This should be able to help as long as there is faint reception. Please note, if you’re in an area with absolutely no reception, then a booster will not function as there is nothing to boost.

You’ll probably need two people, both for testing the signal and adjusting the antenna, as well as fishing cable through your house. Most of the time required for my install was associated with running the wiring.

For testing signal strength, I used the “LTE Discovery” app on Android (https://play.google.com/store/apps/details?id=net.simplyadvanced.ltediscovery)

Feb 182017
 

This is an issue that effects quite a few people and numerous forum threads can be found on the internet by those searching for the solution.

This can occur both when taking manual snapshots of virtual machines when one chooses “Quiesce guest filesystem”, or when using snapshot based backup applications such as vSphere Data Protection (vSphere vDP).

 

For the last couple days, one of my test VMs (Windows Server 2012 R2) has been experiencing this issue and the snapshot has been failing with the following errors:

An error occurred while taking a snapshot: Failed to quiesce the virtual machine.
An error occurred while saving the snapshot: Failed to quiesce the virtual machine.

As always with standard troubleshooting, I restarted the VM, checked for VSS provider errors, and insured that the Windows Services involved with snapshots were in their correct state and configuration. Unfortunately this had no effect, and everything was configured the way it should be.

I also tried to re-install VMWare tools, which had no effect.

PLEASE NOTE: If you experience this issue, you should confirm the services are in their correct state and configuration, as outlined in VMware KB: 1007696. Source: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007696

 

The Surprise Fix:

In the days leading up to the failure when things were running properly, I did notice that the quiesced snapshots for that VM were taking a long time process, but were still functioning correctly before the failure.

This morning during troubleshooting, I went ahead and deleted all the Windows Volume Shadow Copies which are internal and inside of the Virtual Machine itself. These are the shadow copies that the Windows guest operating system takes on it’s own filesystem (completely unrelated to VMware).

To my surprise after doing this, not only was I able to create a quiesced snapshot, but the snapshot processed almost instantly (200x faster than previously when it was functioning).

I’m assuming this was causing a high load for the VMware snapshot to process and a timeout was being hit on snapshot creation which caused the issue. While Windows volume shadow copies are unrelated to VMware snapshots, they both utilize the same VSS (Volume Shadow Copy Service) system inside of windows to function and process. One must also keep in mind that the Windows volume shadow copies will of course be part of a VMware snapshot.

PLEASE NOTE: Deleting your Windows Volume Shadow copies will delete your Windows volume snapshots inside of the virtual machine. You will lose the ability to restore files and folders from previous volume shadow copy snapshots. Be aware of what this means and what you are doing before attempting this fix.

Feb 142017
 

Years ago, HPe released the GL200 firmware for their HPe MSA 2040 SAN that allowed users to provision and use virtual disk groups (and virtual volumes). This firmware came with a whole bunch of features such as Read Cache, performance tiering, thin provisioning of virtual disk group based volumes, and being able to allocate and commission new virtual disk groups as required.

(Please Note: On virtual disk groups, you cannot add a single disk to an already created disk group, you must either create another disk group (best practice to create with the same number of disks, same RAID type, and same disk type), or migrate data, delete and re-create the disk group.)

The biggest thing with virtual storage, was the fact that volumes created on virtual disk groups, could span across multiple disk groups and provide access to different types of data, over different disks that offered different performance capabilities. Essentially, via an automated process internal to the MSA 2040, the SAN would place highly used data (hot data) on faster media such as SSD based disk groups, and place regularly/seldom used data (cold data) on slower types of media such as Enterprise SAS disks, or archival MDL SAS disks.

(Please Note: To use the performance tier either requires the purchase of a performance tiering license, or is bundled if you purchase an HPe MSA 2042 which additionally comes with SSD drives for use with “Read Cache” or “Performance tier.)

 

When the firmware was first released, I had no impulse to try it out since I have 24 x 900GB SAS disks (only one type of storage), and of course everything was running great, so why change it? With that being said, I’ve wanted and planned to one day kill off my linear storage groups, and implement the virtual disk groups. The key reason for me being thin provisioning (the MSA 2040 supports the “DELETE” VAAI function), and virtual based snapshots (in my environment, I require over-commitment of the volume). As a side-note, as of ESXi 6.5, ESXi now regularly unmaps unused blocks when using the VMFS-6 filesystem (if left enabled), which is great for SANs using thin provision that support the “DELETE” VAAI function.

My environment consisted of 2 linear disk groups, 12 disks in RAID5 owned by controller A, and 12 disks in RAID5 owned by controller B (24 disks total). Two weekends ago, I went ahead and migrated all my VMs to the other datastore (on the other volume), deleted the linear disk group, created a virtual disk group, and then migrated all the VMs back, deleted my second linear volume, and created a virtual disk group.

Overall the process was very easy and fast. No downtime is required for this operation if you’re licensed for Storage vMotion in your vSphere environment.

During testing, I’ve noticed absolutely no performance loss using virtual vs linear, except for some functions that utilize the VAAI storage providers which of course run faster on the virtual disk groups since it’s being offloaded to the SAN. This was a major concern for me as block linear based storage is accessed more directly, then virtual disk groups which add an extra level of software involvement between the controllers and disks (block based access vs file based access for the iSCSI targets being provided by the controllers).

Unfortunately since I have no SSDs and no extra room for disks, I won’t be able to try the performance tiering, but I’m looking forward to it in the future.

I highly recommend implementing virtual disk groups on your HPe MSA 2040 SAN!

Feb 082017
 

When running vSphere 6.5 and utilizing a VMFS-6 datastore, we now have access to automatic LUN reclaim (this unmaps unused blocks on your LUN), which is very handy for thin provisioned storage LUNs.

Essentially when you unmap blocks, it “tells” the storage that unused (deleted or moved data) blocks aren’t being used anymore and to unmap them (which decreases the allocated size on the storage layer). Your storage LUN must support VAAI and the “Delete” function.

Most of you have noticed that storage reclaim in the vSphere client has two settings for priority; none, or low.

For those of you who feel daring or want to spice life up a bit, you can increase the priority through the esxcli command. While I can’t recommend this (obviously VMware chose to hide these options due to performance considerations), you can follow these instructions to change the priority higher.

 

To view current settings:

esxcli storage vmfs reclaim config get –volume-label=DATASTORENAME

To set reclaim priority to medium:

esxcli storage vmfs reclaim config set –volume-label=DATASTORENAME –reclaim-priority=medium

To set reclaim priority to high:

esxcli storage vmfs reclaim config set –volume-label=DATASTORENAME –reclaim-priority=high

 

You can confirm these settings took effect by running the command to view settings above, or view the datastore in the storage section of the vSphere client. While the vSphere client will reflect the higher priority setting, if you change it lower and then want to change it back higher, you’ll need to use the esxcli command to bring it up to a higher priority again.

Feb 072017
 

With vSphere 6.5 came VMFS 6, and with VMFS 6 came the auto unmap feature. This is a great feature, and very handy for those of you using thin provisioning on your datastores hosted on storage that supports VAAI.

I noticed something interesting when running the manual unmap command for the first time. It isn’t well documented, but I thought I’d share for those of you who are doing a manual LUN unmap for the first time.

Reason:

Automatic unmap (auto space reclamation) is on, however you want to speed it up or have a large chunk of block’s you want unmapped immediately, and don’t want to wait for the auto feature.

Problem:

I wasn’t noticing any unmaps were occurring automatically and I wanted to free up some space on the SAN, so I decided to run the old command to forcefully run the unmap to free up some space:

esxcli storage vmfs unmap –volume-label=DATASTORENAME –reclaim-unit=200

After kicking it off, I noticed it wasn’t completing as fast as I thought it should be. I decided to enable SSH on the host and took a look at the /var/log/hostd.log file. To my surprise, it wasn’t stopping at a 200 block reclaim, it just kept cycling running over and over (repeatedly doing 200 blocks):

2017-02-07T14:12:37.365Z info hostd[XXXXXXXX] [Originator@XXXX sub=Libs opID=esxcli-fb-XXXX user=root] Unmap: Async Unmapped 200 blocks from volume XXXXXXXX-XXXXXXXX-XXXX-XXXXXXXXX
2017-02-07T14:12:37.978Z info hostd[XXXXXXXX] [Originator@XXXX sub=Libs opID=esxcli-fb-XXXX user=root] Unmap: Async Unmapped 200 blocks from volume XXXXXXXX-XXXXXXXX-XXXX-XXXXXXXXX
2017-02-07T14:12:38.585Z info hostd[XXXXXXXX] [Originator@XXXX sub=Libs opID=esxcli-fb-XXXX user=root] Unmap: Async Unmapped 200 blocks from volume XXXXXXXX-XXXXXXXX-XXXX-XXXXXXXXX
2017-02-07T14:12:39.191Z info hostd[XXXXXXXX] [Originator@XXXX sub=Libs opID=esxcli-fb-XXXX user=root] Unmap: Async Unmapped 200 blocks from volume XXXXXXXX-XXXXXXXX-XXXX-XXXXXXXXX
2017-02-07T14:12:39.808Z info hostd[XXXXXXXX] [Originator@XXXX sub=Libs opID=esxcli-fb-XXXX user=root] Unmap: Async Unmapped 200 blocks from volume XXXXXXXX-XXXXXXXX-XXXX-XXXXXXXXX
2017-02-07T14:12:40.426Z info hostd[XXXXXXXX] [Originator@XXXX sub=Libs opID=esxcli-fb-XXXX user=root] Unmap: Async Unmapped 200 blocks from volume XXXXXXXX-XXXXXXXX-XXXX-XXXXXXXXX
2017-02-07T14:12:41.050Z info hostd[XXXXXXXX] [Originator@XXXX sub=Libs opID=esxcli-fb-XXXX user=root] Unmap: Async Unmapped 200 blocks from volume XXXXXXXX-XXXXXXXX-XXXX-XXXXXXXXX
2017-02-07T14:12:41.659Z info hostd[XXXXXXXX] [Originator@XXXX sub=Libs opID=esxcli-fb-XXXX user=root] Unmap: Async Unmapped 200 blocks from volume XXXXXXXX-XXXXXXXX-XXXX-XXXXXXXXX
2017-02-07T14:12:42.275Z info hostd[XXXXXXXX] [Originator@XXXX sub=Libs opID=esxcli-fb-9XXXX user=root] Unmap: Async Unmapped 200 blocks from volume XXXXXXXX-XXXXXXXX-XXXX-XXXXXXXXX
2017-02-07T14:12:42.886Z info hostd[XXXXXXXX] [Originator@XXXX sub=Libs opID=esxcli-fb-XXXX user=root] Unmap: Async Unmapped 200 blocks from volume XXXXXXXX-XXXXXXXX-XXXX-XXXXXXXXX

That’s just a small segment of the logs, but essentially it just kept repeating the unmap/reclaim over and over in 200 block segments. I waited hours, tried to issue a “CTRL+C” to stop it, however it kept running.

I left it to run overnight and it did eventually finish while I was sleeping. I’m assuming it attempted to unmap everything it could across the entire datastore. Initially I thought this command would only unmap the specified block size.

When running this command, it will continue to cycle in the block size specified until it goes through the entire LUN. Be aware of this when you’re planning on running the command.

Essentially, I would advise not to manually run the unmap command unless you’re prepared to unmap and reclaim ALL your unused allocated space on your VMFS 6 datastore. In my case I did this because I had 4TB of deleted data that I wanted to unmap immediately, and didn’t want to wait for the automatic unmap.

I thought this may have been occurring because the automatic unmap function was on, so I tried it again after disabling auto unmap. The behavior was the same and it just kept running.

 

If you are tempted to run the unmap function, keep in mind it will continue to scan the entire volume (despite what block count you set). With this being said, if you are firm on running this, choose a larger block count (200 or higher) since smaller blocks will take forever (tested with a block size of 1 and after analyzing the logs and rate of unmaps, it would have taken over 3 months to complete on a 9TB array).

Feb 062017
 

Had a nasty little surprise with one of my clients this afternoon. Two days ago I updated their Sophos UTM (UTM220) to version 9.410-6 without any issues.

However, today I started to receive notifications that services were crashing (specifically ACC device agent).

After receiving a few of these, I logged in to check it out. Immediately there was no visible errors on the UTM itself, but after some further digging, I noticed these event logs in the “System Messages” log file:

2017:02:06-17:09:32 mail partitioncleaner[7918]: automatic cleaning for partition /tmp started (inodes: 0/100 blocks: 100/85)

2017:02:06-17:09:32 mail partitioncleaner[7918]: stopping deletion: can’t delete more files

Looks like a potential storage problem? Yes it was, but slightly more complicated.

I enabled SSH on the UTM and issued the “df” command (show’s volume usage), and found that the /tmp volume was 100% full.

Doing a “ls” and “ls -hl”, I found there were 25+ files that were around 235MB in size called: “AV-malware-names-XXXX-XXXXXX”.

Restarting the unit clears those files, however they come back shortly (I noticed it would add one every 5-10 minutes).

After some further digging (still haven’t heard back from Sophos on the support case), I came across some other users experiencing the same issues. While no one found a permanent resolution, they did mention this had to do with the Avira AV engine or possibly the dual scan engine.

Checking the UTM, I noticed that we had the E-Mail scanning configured for dual scan.

Solution (temporary workaround):

I went ahead and configured the E-Mail scanner (the only scanner I had that was using dual scan) to use single scan only. I then restarted the UTM. In my environment the default setting for single scanning is set to “Sophos”.

I am now sitting here with 30 minutes of uptime and absolutely no “AV-malware-names-XXXX-XXXXXX” files created.

I will post an update when I hear back from Sophos support.

Hope this helps someone else!

 

Update (after original post):

I heard back from Sophos support, this is a known bug in 9.410. The current official workaround is to change to single scan and use the AVIRA engine instead of the Sophos engine.

Update #2:

Received notification this morning of a new firmware update available (Version: 9.411003 – Maintenance Release). While I haven’t installed it, it appears from the Bugfixes notes that it was released to fix this issue:

 Fix [NUTM-6804]: [AWS] Update breaks HVM standalone installations
Fix [NUTM-6747]: [Email] SAVI scanner coredumps permanently in MailProxy after update to 9.410
Fix [NUTM-6802]: [Web] New coredumps from httpproxy after update to v9.410

Update #3:

I noticed that this bug was interrupting some mailflow on my Sophos UTM, as well as some of my clients. I went ahead and as an emergency situation, installed 9.411-3.

Things were fine for around 10 hours until I started to receive notification of the HTTP proxy failing and requiring restart. Logging in to the UTM, it was very unresponsive, sometimes completely unresponsive for around 10 minutes. Web browsing was not functioning at all on the internal network behind the UTM.

This issue still hasn’t been resolved. Hopefully we see a stable working fix sometime soon.