Aug 232019
 
Stephen Wagner Image

Just a reminder to my readers that in addition to IT consulting services, I also provide emergency IT consulting services. I’m available weekends and holidays, and I’m also available to travel.

If you already haven’t figured it out (from all the links on my page), you can find out more information here https://www.stephenwagner.com/hire-stephen-wagner-it-services/, which includes what I provide, how billing works, my rates, and more.

Numerous readers have utilized my services in the past, and some do on a frequent and regular basis.

Remote IT Consulting

In a bind and need help fast? Feel free to reach out. I can connect remotely, provide assistance with issues, implementations, migrations, hardware, software, licensing, and pretty much anything that’s been discussed in this blog or relating to Information Technology.

On-Site IT Consulting Services

I’m also available for assistance on-site and in person, with the ability to travel. Have a remote site that has a major IT issue? Give me a shout!

Get in Touch

As mentioned before, click here for more information: https://www.stephenwagner.com/hire-stephen-wagner-it-services/

Don’t hesitate to reach out, even competitors have utilized my services!

Aug 202019
 
Looking east from the summit at Grotto Mountain in Canmore Alberta

Earlier this month, we decided to hike and climb Grotto Mountain. Grotto Mountain is just outside of Canmore, Alberta, with it’s trail head starting right by the Alpine club. This was my 3rd time doing the hike, and 1st for my friend. This was however the first time I’ve done the complete loop, ascending the ACC route, and descending the hard route.

While we completed the loop in a clockwise manner, I highly recommend against this. From the summit down, it was extremely difficult to find the trail, even with a downloaded map and GPS. Not only will you get lost, but it’s also incredibly difficult (probably one of the most difficult descents I’ve done simply because I kept slipping).

I’d highly recommend doing the loop in a counter-clockwise method. While this trail is “safe”, it is difficult and challenging requiring lots of stamina and cardio work. Things can get a little risky on the hard route.

AllTrails Link: https://www.alltrails.com/trail/canada/alberta/grotto-mountain-trail

My AllTrails Recording: https://www.alltrails.com/explore/recording/recording-aug-04-06-07-pm–3

Looking east from the summit at Grotto Mountain in Canmore Alberta
Grotto Mountain Summit view East

Grotto Mountain is an ascent up to an altitude of around 2,870m (9416ft), with beautiful views of Canmore, Alberta and other mountains. I completed this trek with my usual hiking buddy Elisha!

Selfie of Stephen Wagner and Elisha Comeau standing on Grotto Mountain
Stephen Wagner and Elisha Comeau – Grotto Mountain
Stephen Wagner standing on Grotto Mountain Summit
Stephen Wagner on Grotto Mountain Summit
Elisha Comeau standing on Grotto Mountain Summit
Elisha Comeau on Grotto Mountain Summit
Summit view from Grotto Mountain
Summit view from Grotto Mountain

And of course, below is a picture of the mountain from Canmore, Alberta. The summit is actually the peak/summit on the right side of the mountain, the left is the lower fake summit.

As I mentioned above, it’s very challenging even having done it a few times before. The ACC route is a nice long slow climb and descent, while the hard route is pretty much straight up and down.

Along the hard route we did see some wild life like mountain goats, but they stayed far away. I’ve never seen bears on this hike, however I believe there may be a risk at the bottom, as well as up to the point of the top of the tree line.

Grotto Mountain Hike Pictures

Stay safe, be bear aware, and always make sure you always do hikes like this with a friend!

Aug 122019
 
DS1813+

Around a month ago I decided to turn on and start utilizing NFS v4.1 (Version 4.1) in DSM on my Synology DS1813+ NAS. As most of you know, I have a vSphere cluster with 3 ESXi hosts, which are backed by an HPe MSA 2040 SAN, and my Synology DS1813+ NAS.

The reason why I did this was to test the new version out, and attempt to increase both throughput and redundancy in my environment.

If you’re a regular reader you know that from my original plans (post here), and than from my issues later with iSCSI (post here), that I finally ultimately setup my Synology NAS to act as a NFS datastore. At the moment I use my HPe MSA 2040 SAN for my hot storage, and I use the Synology DS1813+ for cold storage. I’ve been running this way for a few years now.

Why NFS?

Some of you may ask why I chose to use NFS? Well, I’m an iSCSI kinda guy, but I’ve had tons of issues with iSCSI on DSM, especially MPIO on the Synology NAS. The overhead was horrible on the unit (result of the lack of hardware specs on the NAS) for both block and file access to iSCSI targets (block target, vs virtualized (fileio) target).

I also found a major issue, where if one of the drives were dying or dead, the NAS wouldn’t report it as dead, and it would bring the iSCSI target to a complete halt, resulting in days spending time finding out what’s going on, and then finally replacing the drive when you found out it was the issue.

After spending forever trying to tweak and optimize, I found that NFS was best for the Synology NAS unit of mine.

What’s this new NFS v4.1 thing?

Well, it’s not actually that new! NFS v4.1 was released in January 2010 and aimed to support clustered environments (such as virtualized environments, vSphere, ESXi). It includes a feature called Session trunking mechanism, which is also known as NFS Multipathing.

We all love the word multipathing, don’t we? As most of you iSCSI and virtualization people know, we want multipathing on everything. It provides redundancy as well as increased throughput.

How do we turn on NFS Multipathing?

According to the VMware vSphere product documentation (here)

While NFS 3 with ESXi does not provide multipathing support, NFS 4.1 supports multiple paths.


NFS 3 uses one TCP connection for I/O. As a result, ESXi supports I/O on only one IP address or hostname for the NFS server, and does not support multiple paths. Depending on your network infrastructure and configuration, you can use the network stack to configure multiple connections to the storage targets. In this case, you must have multiple datastores, each datastore using separate network connections between the host and the storage.


NFS 4.1 provides multipathing for servers that support the session trunking. When the trunking is available, you can use multiple IP addresses to access a single NFS volume. Client ID trunking is not supported.

So it is supported! Now what?

In order to use NFS multipathing, the following must be present:

  • Multiple NICs configured on your NAS with functioning IP addresses
  • A gateway is only configured on ONE of those NICs
  • NFS v4.1 is turned on inside of the DSM web interface
  • A NFS export exists on your DSM
  • You have a version of ESXi that supports NFS v4.1

So let’s get to it! Enabling NFS v4.1 Multipathing

  1. First log in to the DSM web interface, and configure your NIC adapters in the Control Panel. As mentioned above, only configure the default gateway on one of your adapters.Synology Multiple NICs Configured Screenshot
  2. While still in the Control Panel, navigate to “File Services” on the left, expand NFS, and check both “Enable NFS” and “Enable NFSv4.1 support”. You can leave the NFSv4 domain blank.Enabling NFSv4.1 on Synology DSM
  3. If you haven’t already configured an NFS export on the NAS, do so now. No further special configuration for v4.1 is required other than the norm.
  4. Log on to your ESXi host, go to storage, and add a new datastore. Choose to add an NFS datastore.
  5. On the “Select NFS version”, select “NFS 4.1”, and select next.Selecting the NFS version on the Add Datastore Dialog box on ESXi
  6. Enter the datastore name, the folder on the NAS, and enter the Synology NAS IP addresses, separated by commas. Example below:New NFS Datastore details and configuration on ESXi dialog box
  7. Press the Green “+” and you’ll see it spreads them to the “Servers to be added”, each server entry reflecting an IP on the NAS. (please note I made a typo on one of the IPs).List of Servers/IPs for NFS Multipathing on ESXi Add Datastore dialog box
  8. Follow through with the wizard, and it will be added as a datastore.

That’s it! You’re done and are now using NFS Multipathing on your ESXi host!

In my case, I have all 4 NICs in my DS1813+ configured and connected to a switch. My ESXi hosts have 10Gb DAC connections to that switch, and can now utilize it at faster speeds. During intensive I/O loads, I’ve seen the full aggregated network throughput hit and sustain around 370MB/s.

After resolving the issues mentioned below, I’ve been running for weeks with absolutely no problems, and I’m enjoying the increased speed to the NAS.

Additional Important Information

After enabling this, I noticed that RAM and Memory usage had drastically increased on the Synology NAS. This would peak when my ESXi hosts would restart. This issue escalated to the NAS running out of memory (both physical and swap) and ultimately crashing.

After weeks of troubleshooting I found the processes that were causing this. While the processes were unrelated, this issue would only occur when using NFS Multipathing and NFS v4.1. To resolve this, I had to remove the “pkgctl-SynoFinder” package, and disable the services. I could do this in my environment because I only use the NAS for NFS and iSCSI. This resolved the issue. I created a blog post here to outline how to resolve this. I also further optimized the NAS and memory usage by disabling other unneeded services in a post here, targeted for other users like myself, who only use it for NFS/iSCSI.

Leave a comment and let me know if this post helped!

Aug 092019
 
Sophos UTM with SFP Modules Picture

Today (August 9th, 2019, starting in the early morning) I noticed that numerous Sophos UTM firewalls were sending the notification “The spam filter daemon is unable to reach the database servers via HTTP. Please make sure that the device is able to send HTTP (TCP port 80) requests to the Internet. You may have to allow such traffic on upstream devices.”.

Everything is still functioning and upon troubleshooting I noticed that nothing had been changed, nor was broken. I believe this is a service outage of some type.

This issue has also been reported by numerous other users here: https://community.sophos.com/products/unified-threat-management/f/mail-protection-smtp-pop3-antispam-and-antivirus/114516/getting-spam-filter-cannot-query/411525

I will be updating this post with more information, leave a comment if you have information, or if the issue is also happening to you!

Aug 092019
 
IIS Logo Image

You may find yourself unable to download attachments on an e-mail message you received on your Android or Apple iPhone from your Microsoft Exchange Server. In my case, this presented a “Unable to download.” with a retry option. Retrying would not work.

If the attachment is larger (over 10MB), this is most likely due to a limit enforced on the Activesync site in IIS on your Exchange Server. In this post I’m going to tell you why this happens, and how to fix it!

The Problem

Microsoft Exchange uses IIS (Internet Information Server) for numerous services including ActiveSync. ActiveSync provides the connectivity to your mobile device for your Exchange access.

IIS has numerous limits configured to stop massive bogus requests, reduce DDOS attacks, and other reasons.

The Fix

To resolve this and allow the attachment to download, we need to modify two configuration values inside of the web.config file on IIS.

Below are the values we will be modifying:

  • MaxDocumentDataSize – Maximum file (message) data size for transfer. “Sets the maximum data size that we will fetch (range or othewise)”
  • maxRequestLength – “Specifies the limit for the input stream buffering threshold, in KB. This limit can be used to prevent denial of service attacks that are caused, for example, by users posting large files to the server. The default is 4096 KB.” (as per here)

These settings are configured in the following file:

C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\Sync\web.config

Before modifying the variables, please make a copy or backup of the web.config file so you can restore.

After you make a backup, open the file in notepad (right click -> run as administrator), and open the web.config file.

Simply search for the two values listed above, and change them. In my case, I tripled the “MaxDocumentDataSize”, and the “maxRequestLength” values. Examples from my “web.config” file are below:

add key="MaxDocumentDataSize" value="30720000"
httpRuntime maxRequestLength="30720" fcnMode="Disabled"

After changing these, run the following command from an elevated (as administrator) command prompt to restart IIS:

iisreset

And bam, you’re good to go!

Jul 312019
 
Rogers RCS on Samsung Galaxy S9+ Settings Screenshot Confirming RCS

This morning I had two surprises, the first being the not-so monthly security update, and the second being that the Samsung Galaxy S9+ messaging app showed a notification about turning on RCS (Rich Communication Services) messaging.

So as of today, RCS messaging is officially enabled on Samsung Galaxy S9+ phones on Rogers in Canada. Please note that this option did NOT appear on other Samsung S8/S8+ phones on my corporate account, and only the S9+.

This function has also probably been enabled on newer Samsung Galaxy devices as well. Please leave a comment if you can confirm.

Turning on RCS

To turn on RCS, open the messaging app. You’ll notice a “notification bubble” on the three-dot menu button.

Click on this, going to “Settings”, you’ll be able to turn on what Samsung calls “Chat Service”. Tap on this, and it will have to register on the network (since RCS is a service provided by your cell network).

RCS Settings

Once you register RCS on the network, a new settings button will appear called “Chat Settings”.

Rogers RCS on Samsung Galaxy S9+ Menu Screenshot
Rogers RCS on Samsung Galaxy S9+ Menu

Once you go in to “Chat Settings” you’ll see the following options, confirming it is RCS.

Rogers RCS on Samsung Galaxy S9+ Settings Screenshot Confirming RCS
Rogers RCS on Samsung Galaxy S9+ Settings

As of right now I can’t really test this, because all the other phones on my corporate account are only S8+ or S8 variants. I also don’t know anyone that is currently using RCS, so I’ll have to wait to test it out.

Jul 312019
 

If you’re like me and use a Synology NAS as an NFS or iSCSI datastore for your VMware environment, you want to optimize it as much as possible to reduce any hardware resource utilization.

Specifically we want to disable any services that we aren’t using which may use CPU or memory resources. On my DS1813+ I was having issues with a bug that was causing memory overflows (the post is here), and while dealing with that, I decided to take it a step further and optimize my unit.

Optimize the NAS

In my case, I don’t use any file services, and only use my Synology NAS (Synology DS1813+) as an NFS and iSCSI datastore. Specifically I use multipath for NFSv4.1 and iSCSI.

If you don’t use SMB (Samba / Windows File Shares), you can make some optimizations which will free up substantial system resources.

Disable and/or uninstall unneeded packages

First step, open up the “Package Center” in the web GUI and either disable, or uninstall all the packages that you don’t need, require, or use.

To disable a package, select the package in Package Center, then click on the arrow beside “Open”. A drop down will open up, and “Disable” or “Stop” will appear if you can turn off the service. This may or may not be persistent on a fresh boot.

To uninstall a package, select the packet in Package Center, then click on the arrow beside “Open”. A drop down will open up, and “Uninstall” will appear. Selecting this will uninstall the package.

Disable the indexing service

As mentioned here, the indexing service can consume quite a bit of RAM/memory and CPU on your Synology unit.

To stop this service, SSH in to the unit as admin, then us the command “sudo su” to get a root shell, and finally run this command.

synoservice --disable pkgctl-SynoFinder

The above command will probably not persist on boot, and needs to be ran each fresh boot. You can however uninstall the package with the command below to completely remove it.

synopkg uninstall SynoFinder

Doing this will free up substantial resources.

Disable SMB (Samba), and NMBD

I noticed that both smbd and nmbd (Samba/Windows File Share Services) were consuming quite a bit of CPU and memory as well. I don’t use these, so I can disable them.

To disable them, I ran the following command in an SSH session (remember to “sudo su” from admin to root).

synoservice --disable nmbd
synoservice --disable samba

Keep in mind that while this should be persistent on boot, it wasn’t on my system. Please see the section below on how to make it persistent on booth.

Disable thumbnail generation (thumbd)

When viewing processes on the Synology NAS and sorting by memory, there are numerous “thumbd” processes (sometimes over 10). These processes deal with thumbnail generation for the filestation viewer.

Since I’m not using this, I can disable it. To do this, we either have to rename or delete the following file. I do recommend making a backup of the file.

/var/packages/FileStation/target/etc/conf/thumbd.conf

I’m going to rename it so that the service daemon can’t find it when it initializes, which causes the process not to start on boot.

cd /var/packages/FileStation/target/etc/conf/
mv thumbd.conf thumbd.conf.bak

Doing the above will stop it from running on boot.

Make the optimizations persistent on boot

In this section, I will show you how to make all the settings above persistent on boot. Even though I have removed the SynoFinder package, I still will create a startup script on the Synology NAS to “disable” it just to be safe.

First, SSH in to the unit, and run “sudo su” to get a root shell.

Run the following commands to change directory to the startup script, and open a text editor to create a startup script.

cd /usr/local/etc/rc.d/
vi speedup.sh

While in the vi file editor, press “i” to enter insert mode. Copy and paste the code below:

case "$1" in
    start)
                echo "Turning off memory garbage"
                        synoservice --disable nmbd
                        synoservice --disable samba
                        synoservice --disable pkgctl-SynoFinder
                        ;;
    stop)
                        echo "Pertend we care and are turning something on"
                        ;;
        *)
        echo "Usage: $1 {start|stop}"
                exit 1
esac
exit 0

Now press escape, then type “:wq” and hit enter to save and close the vi text editor. Run the following command to make the script executable.

chmod 755 speedup.sh

That’s it!

Conclusion

After making the above changes, you should see a substantial performance increase and reduction in system resources!

In the future I plan on digging deeper in to optimization as I still see other services I may be able to trim down, after confirming they aren’t essential to the function of the NAS.

Feel like you can add anything? Leave a comment!

Jul 312019
 

Once I upgraded my Synology NAS to DSM 6.2 I started to experience frequent lockups and freezing on my DS1813+. The Synology DS1813+ would become unresponsive and I wouldn’t be able to SSH or use the web GUI to access it. In this state, NFS sometimes would become unresponsive.

When this occured, I would need to press and hold the power button to force it to shutdown, or pull the power. This is extremely risky as it can cause data corruption.

I’m currently running DSM 6.2.2-24922 Update 2.

The cause

This occurred for over a month until it started to interfere with ESXi hosts. I also noticed that the issue would occur when restarting any of my 3 ESXi hosts, and would definitely occur if I restarted more than one.

During the restarting, while logged in to the web GUI and SSH, I was able to see that the memory (RAM) usage would skyrocket. Finally the kernel would panic and attempt to reduce memory usage once the swap file had filled up (keep in mind my DS1813+ has 4GB of memory).

Analyzing “top” as well as looking at processes, I noticed the Synology index service was causing excessive memory and CPU usage. On a fresh boot of the NAS, it would consume over 500MB of memory.

The fix

In my case, I only use my Synology NAS for an NFS/iSCSI datastore for my ESXi environment, and do not use it for SMB (Samba/File Shares), so I don’t need the indexing service.

I went ahead and SSH’ed in to the unit, and ran the following commands to turn off the service. Please note, this needs to be run as root (use “sudo su” to elevate from admin to root).

synoservice --disable pkgctl-SynoFinder

While it did work, and the memory was instantly freed, the setting did not stay persistant on boot. To uninstalling the Indexing service, run the following command.

synopkg uninstall SynoFinder

Doing this resolved the issue and freed up tons of memory. The unit is now stable.

Update – August 16th, 2019

My Synology NAS has been stable since I applied the fix, however after an uptime of a few weeks, I noticed that when restarting servers, the memory usage does hike up (example, from 6% to 46%). However, with the fixes applied above, the unit is stable and no longer crashes.

Jul 192019
 
King Creek Ridge Summit

Sunday (July 14th, 2019) we had the opportunity to hike King Creek Ridge. I’d considering this hike a moderate hike, very steep, and can be completed in a short amount of time. Taking the traditional route to the summit, it took us around just under an hour. This hike is located in Kananaskis Country, and there is no cell reception anywhere on the trail.

There are other trails (and alternate trails) along this path. I do not recommend taking these or attempting these. Please read below to find out why.

AllTrails Link: https://www.alltrails.com/trail/canada/alberta/king-creek-trail

King Creek Ridge Summit View
King Creek Ridge Summit View

King Creek Ridge is a steep ascent to an altitude of around 2,400m (7,874ft), so it’ll require some cardio. I completed this hike with my friend Elisha, and her step-dad Tim!

Stephen Wagner and Elisha posing on King Creek Ridge
Stephen Wagner and Elisha on King Creek Ridge
King Creek Ridge Valley View
King Creek Ridge Valley

On this hike, we met a nice group of other hikers who mentioned that they read in a book (that was 15 years old), about a loop on this trail. Once at the summit, they planned to go down the Mount Hood route to complete the loop. Using the map I had downloaded to my mobile, I actually had a copy of the Mount Hood route, so we figured we’d attempt it as well.

Mount Hood Route Detour

IMPORTANT: DO NOT ATTEMPT THE MOUNT HOOD ROUTE! IT IS AN OLD ROUTE AND THERE IS NO VISIBLE TRAIL. IT’S FULL OF DENSE BUSH, LIMITED VISIBILITY, AND HAS WILDLIFE (BEARS).

Once we had our time at the summit and relaxed (we flew up there pretty quick), we decided to try and locate the path down to the valley to complete the Mount Hood route. We located a small channel that was very steep and full of loose rocks (extremely dangerous). After navigating down, we noticed that an avalanche or weather related incident had damaged the path and it was simply a cliff. We proceeded to find a different way to go down in to the valley after doing a big of climbing down.

King Creek Ridge to Mount Hood Route
King Creek Ridge to Mount Hood Route

After the dangerous part, we found a steep path down (shown above) that we used to get in to the valley. Unfortunately this was short lived. Once getting in to the valley, we approached bush that was ~5 ft high, and numerous places to be ambushed by bears. We had to use boulders to throw in to the bushes ahead to make sure there were no bears further up (also using regular bursts of the air horn).

Finally once we made it to the creek, we realized using the GPS that the Mount Hood Route was extremely old and grown over. There was no visible trail for Mount Hood Route. We continued along the creek (walking in/over it) for some time until it became so difficult and dangerous that we decided we had to turn around and go back.

Back to King Creek Ridge from Mount Hood Route
Back to King Creek Ridge from Mount Hood Route

The hike/climb back up to the summit of the King Creek Ridge hike was grueling. It was one of the steepest hikes I’ve done, and finished with sections that we actually had to climb (using our hands) to get back to the King Creek Ridge summit.

Back on to King Creek Ridge

So the detour, while extremely dangerous and unnerving, actually made the hike as long as we had originally wanted it, and as fun as we had hoped, so it turned out pretty good (after we realized we weren’t going to be stuck in the valley all night, lol).

We finally descended down the way we had originally came up, and all in all the day turned out to be amazing! The hiking trail actually was extremely busy later on in the afternoon by the time we went down.

King Creek Ridge Hike Pictures

Below are pictures from the hike, please click to enter the attachment page, and then click again to enlarge. The below are low quality thumbnails, when you click on the image it will open the high-resolution image.

Stay safe, be bear aware, and always verify trails exist before you head out on to them!

Jul 122019
 

Last week (July 3rd, 2019), I hiked and climbed Mount Lawrence Grassi with my friend Elisha. It was a very aggressive hike, ascending very quickly to an altitude of 2,685 meters (8,809 feet).

It took us around 7 hours and 45 minutes, with a total moving time of 4 hours and 43 minutes. We completed a total elevation gain of 1,177m. My cardio has been horrible this year as I’ve been focusing on strength training so I slowed us down a little bit.

Picture gallery is below of the hike.

I’d only recommend this hike only for advanced hikers as the last 100 feet can be extremely dangerous (risk of slipping, sliding off cliff on both sides). Cell phone reception (for emergencies) is available for most of the hike.

Stephen Wagner Posing on Mount Lawrence Grassi while Hiking
Stephen Wagner on Mount Lawrence Grassi

We’ve had issues with weather this year, but fortunately for us we chose the perfect day. It was a bit chilly at the top, but it turned out perfectly! The views from the summit were amazing!

View from Summit at Mount Lawrence Grassi

Other than the sketchy 100ft at the top, the only other issue we had was a bear encounter on the way back to the vehicle. Once we got back to the bottom of the mountain, starting our 2km walk to the vehicle, we crossed paths with a black bear. We immediately prepped our bear counter-measures (we always carry bear mace, bear bangers, air horn, etc), however two chirps of the air horn caused it to retreat in to the forest.

AllTrails Link: https://www.alltrails.com/trail/canada/alberta/mount-lawrence-grassi

Mount Lawrence Grassi Hike Pictures

Below are pictures from the hike, please click to enlarge. The below are low quality thumbnails, when you click on the image it will open the high-resolution image.

The best photos are at the end of the album when we reached the summit!

Stay safe and be bear aware!