Aug 192018
 

I finally got around to mounting my Wilson weBoost Home 4G Cell Phone Booster Antenna on the roof. Here’s some pictures of the completed install. I’ve had this booster for a while and it’s worked great, however some new cell towers went up in the area, and I wanted to stop using the window mount and re aim the antenna.

Wilson weBoost Home 4G Cell Phone Booster Roof Outdoor Antenna

Wilson weBoost Home 4G Cell Phone Booster Roof Outdoor Antenna

For those of you wanting to read my original post on the Wilson weBoost Home 4G Cell Phone Booster Kit, installation, and a review, you can find it at https://www.stephenwagner.com/2017/06/01/cellmobile-phone-reception-issues-resolve-with-a-wilson-amplifier-cell-booster/.

The house that I live in, actually had a roof mounted satellite dish that was no longer in use (used before the provider ran coax in the area). The dish, roof mount, and coax were all in place, however the coax was cut so I couldn’t re-use it.

I was able to remove 2 of the bolts on the satelite dish to remove it from the pole mount, and proceeded to install the antenna on the pole using the outdoor mounting kit included with the cell booster. I was extremely pleased with the install.

See below for more pics:

Roof mounted Wilson weBoost Home 4G Cell Phone Booster Kit

Roof mounted Wilson weBoost Home 4G Cell Phone Booster Kit

Roof mounted Wilson weBoost Home 4G Cell Phone Booster Kit Cabling

Roof mounted Wilson weBoost Home 4G Cell Phone Booster Kit Cabling

Roof mounted Antenna pole mount

Roof mounted Antenna pole mount

The cabling goes through the pole, down to the eavestrough where I have it zip-stripped (yet elevated) along the roof until I get to the house’s siding. I was able to tuck it in the corner siding down to the wiring access panel for the house, then into the house through the hole.

After mounting it, it took around 30 minutes to aim it with the assistance of the “LTE Discover” Android app (available at https://play.google.com/store/apps/details?id=net.simplyadvanced.ltediscovery). Remember, when aiming your antenna, it’s important to unplug your booster for 5-10 seconds for it to fully reset for it to function with the new antenna position.

Again, make sure you check out my original post and review at https://www.stephenwagner.com/2017/06/01/cellmobile-phone-reception-issues-resolve-with-a-wilson-amplifier-cell-booster/!

Aug 182018
 
CentOS Logo

Let’s say that you’re hosting someone’s equipment and they start to abuse their connection speed. Let’s say that you’re limited in your bandwidth, and you want to control your own bandwidth to make sure you don’t max out your own internet connection. You can take care of both of these problems by building your own traffic shaping network control device using CentOS and using the “tc” linux command.

In this post I’m going to explain what traffic shaping is, why you’d want to use traffic shaping, and how to build a very basic traffic shaping device to control bandwidth on your network.

What is traffic shaping

Traffic shaping is when one attempts to control a connection in their network to prioritize, control, or shape traffic. This can be used to control either bandwidth or packets. In this example we are using it to control bandwidth such as upload and download speeds.

Why traffic shaping

For service providers, when hosting customer’s equipment, the customer may abuse their connection or even max it out legitimately. This can put a halt on the internet connection if you share it with them, or cause bigger issues if it’s shared with other customers. In this example, you would want to implement traffic shaping to allot only a certain amount of bandwidth so they wouldn’t bring the internet connection or network to a halt.

For normal people (or a single business), as fast as the internet is today, it’s still very easy to max your connection out. When this happens you can experience packet loss, slow speeds, and interruption of services. If you host your own servers this can cause even a bigger issue with interruption of those services as well. You may want to limit your own bandwidth to make sure that you don’t bring your internet to a halt, and save some for other devices and/or users.

Another reason is just to implement basic QoS (Quality of Service) across your network, to keep usage and services in harmony and eliminate any from hogging the network connections up.

How to build your own basic traffic shaping device with CentOS and tc

In this post we will build a very simple traffic shaping device that limits and throttles an internet connection to a defined upload and download speed that we set.

You can do this with a computer with multiple NICs (preferably one NIC for management, one NIC for internet, and one NIC for network and/or the hosts to be throttled). If you want to get creative, there are also a number of physical network/firewall appliances that are x86 based, that you can install Linux on. These are very handy as they come with many NICs.

When I set this up, I used an old decommissioned Sophos UTM 220 that I’ve had sitting around doing nothing for a couple years (pic below). The UTM 220 provides 8 NICs, and is very easy to install Linux on to.

Sophos UTM 220 Running CentOS Linux

Sophos UTM 220 Running CentOS Linux

Please Note: The Sophos UTM 220 is just a fancy computer in a 1U rack mounted case with 8 NICs. All I did was install CentOS on it like a normal computer.

Essentially, all we’ll be doing is installing CentOS Linux, installing “tc”, configuring the network adapters, and then configuring a startup script. In my example my ISP provides me 174Mbps download, and 15Mbps upload. My target is to throttle the connection to 70Mbps download, and 8Mbps upload. I will allow the connection to burst to 80Mbps down, and 10Mbps up.

To get started:

  1. Install CentOS on the computer or device. The specifics of this are beyond the scope of this document, however you’ll want to perform a minimal install. This device is strictly acting as a network device, so no packages are required other than the minimal install option.
  2. During the CentOS install, only configure your main management NIC. This is the NIC you will use to SSH to, control the device, and update the device. No other traffic will pass through this NIC.
  3. After the install is complete, run the following command to enable ssh on boot:
    chkconfig sshd on
  4. Install “tc” by running the command:
    yum install tc
  5. Next, we’ll need to locate the NIC startup scripts for the 2 adapters that will perform the traffic shaping. These adapters are the internet NIC, and the NIC for the throttled network/hosts. Below is an example of one of the network startup scripts. You’re NIC device names will probably be different.
    /etc/sysconfig/network-scripts/ifcfg-enp2s0
  6. Now you’ll need to open the file using your favorite text editor and locate and set ONBOOT to no as shown below. You can ignore all the other variables. You’ll need to repeat this for the 2nd NIC as well.
    TYPE=Ethernet
    PROXY_METHOD=none
    BROWSER_ONLY=no
    BOOTPROTO=dhcp
    DEFROUTE=yes
    IPV4_FAILURE_FATAL=no
    IPV6INIT=yes
    IPV6_AUTOCONF=yes
    IPV6_DEFROUTE=yes
    IPV6_FAILURE_FATAL=no
    IPV6_ADDR_GEN_MODE=stable-privacy
    NAME=enp2s0
    UUID=xxxxxxxx-xxxx-xxx-xxxx-xxxxxxxxxxxx
    DEVICE=enp2s0
    ONBOOT=no
  7. Now we can configure the linux startup script to configure a network bridge between the two NICs above, and then configure the traffic shaping rules with tc. Locate and open the following file for editing:
    /etc/rc.d/rc.local
  8. Append the following text to the rc.local file:
    # Lets make that bridge
    brctl addbr bridge0
    
    # Lets add those NICs to the bridge
    brctl addif bridge0 enp5s0
    brctl addif bridge0 enp2s0
    
    # Confirm no IP set to NICs that are shaping
    ifconfig enp5s0 0.0.0.0
    ifconfig enp2s0 0.0.0.0
    
    # Bring the bridge online
    ifconfig bridge0 up
    
    # Clear out any existing tc policies
    tc qdisc del dev enp2s0 root
    tc qdisc del dev enp5s0 root
    
    # Configure new traffic shaping policies on the NICs
    # Set the upload to 8Mbps and burstable to 10mbps
    tc qdisc add dev enp2s0 root tbf rate 8mbit burst 10mbit latency 50ms
    # Set the download to 70Mbps and burstable to 80Mbps
    tc qdisc add dev enp5s0 root tbf rate 70mbit burst 80mbit latency 50ms
    
  9. Restart the linux box:
    shutdown -r now
  10. You now have a traffic shaping network device!

Final Thoughts

Please note that normally you would not place the script in the rc.local file, however we wanted something quick and simple. The script may not survive in the rc.local file when updates/upgrades are applied against on the Linux install, so keep this in mind. You’ll also need to test to make sure that you are throttling in the correct direction with the 2 NICs. Make sure you test this setup and allow time to confirm it’s working before putting it in a production network.

Aug 182018
 
VMware vSphere Mobile Watchlist Logo

Did you know that you can monitor and manage your VMware vSphere environment (ESXi hosts, cluster, and VMs) remotely with the “VMware vSphere Mobile Watchlist” app on your Android phone? Well, you can!

Download link: https://play.google.com/store/apps/details?id=com.vmware.beacon

The VMware vSphere Mobile Watchlist (VMware Watchlist) Android App

For some time now, I’ve been using this neat little app from VMware (available for download here) to monitor and manage my vSphere cluster remotely. You can use the app while directly on your LAN, or via VPN (I use it with OpenVPN to connect to my Sophos UTM). I’ve even used it while on airplanes using the on board in-flight WiFi.

The reason why I’m posting about this, is because I’ve never actually heard anyone talk about the app (which I find strange), so I’m assuming others aren’t aware of it’s existence as well.

The app runs extremely well on my Samsung S8+, Samsung S9+, and Samsung Tab E LTE tablet. I haven’t run in to any issues or app crashes.

Let’s take a look at the app

vSphere Mobile Watchlist Login Prompt

vSphere Mobile Watchlist Login Prompt

The above screen is where you initially log in. I use my Active Directory credentials (since I have my vCenter server integrated with AD).

vSphere Mobile Watchlist Hosts and VM list

vSphere Mobile Watchlist Hosts and VMs

In the default view (shown above), you can view a brief summary of your ESXi hosts, as well as a list of virtual machines running.

vSphere Mobile Watchlist Host Information

vSphere Mobile Watchlist Host Information

After selecting an ESXi host, you can view the hosts resources, details, related objects, as well as flip over to view host options.

vSphere Mobile Watchlist Host Options

vSphere Mobile Watchlist Host Options

Under host options, you can Enter Maintenance mode, reboot the host, shutdown the host, disconnect the host, or view the hosts’ sensor data.

vSphere Mobile Watchlist Host Sensor List and Fan Data

vSphere Mobile Watchlist Host Sensor Data (Fans)

Checking the HPe Proliant DL360p Gen8 fan sensor data with VMware Watchlist.

vSphere Mobile Watchlist Host Sensor Data (Temperature Sensor List)

vSphere Mobile Watchlist Host Sensor Data (Temperature)

Checking the HPe Proliant DL360p Gen8 temperature sensor data with VMware Watchlist. While not shown above, you can select individual items to pull the actual temperature values. Please Note that the temperature values are missing a decimal (Example: 2100 = 21.00 Celsius).

vSphere Mobile Watchlist VM Information

vSphere Mobile Watchlist VM Information View

When selecting a VM (Virtual Machine) from the default view, you can view the VM’s Resources (CPU, Memory, and Storage), Details (IP Addresses, DNS hostnames, Guest OS, VMWare Tools Status), related objects, and a list of other VMs running on the same host.

vSphere Mobile Watchlist VM Options

vSphere Mobile Watchlist VM Options

Flipping over to the VM options, we have the ability to power off, suspend, reset, shutdown, or gracefully restart the VM. We also have some snapshot functionality to take a snapshot, or manage VM snapshots.

Additional Notes

In my environment I have two HPe DL360p Gen8 Servers and the sensor data is fully supported (I used the HPe custom ESXi install image which includes host drivers).

Aug 152018
 

As of yesterday, Samsung Pay is now available in Canada on your Samsung Gear S3 (and Gear Sport) smartwatch. You’ll need to update the Samsung Gear up to the new Galaxy Wearable app to enable the feature.

Samsung Pay for Samsung Gear S3 and Gear Sport

For some time users have been frustrated that while Samsung Pay has been available in Canada on your mobile devices, it hasn’t been available on their Samsung Gear smartwatches. This is no longer the case. While it has been released, some users are still reporting issues when trying to activate, such as the “Watch is not supported” error.

To get it working:

  1. Make sure your firmware is up to date on your watch using the Samsung Gear app (or the new Samsung Wearable app).
  2. Make sure your app is up to date. Samsung Gear should update to the new name of Samsung Wearable.
  3. Open “Samsung Wearable”, scroll down and look for “Samsung Pay” on the “Info” or “Settings” tab. Tap on this.
  4. The “Samsung Pay (Plug-in) should now download. Wait for it to launch.
  5. When opening Samsung Pay in the Samsung Wearable app, you may receive the error “Watch is not supported”.
  6. Go to “About Gear” under the “Settings” tab, and perform a full manual backup of your Samsung Gear. As soon as this completed, I received the notification on my watch on how to configure Samsung Pay.
  7. Press and hold the “Back” button on the watch, and follow the instructions. The Samsung Pay app should then install on your Samsung Gear watch.

I swear by Samsung pay (and use it with a number of my cards and loyalty programs). It’s awesome that it’s finally been released in Canada!

Aug 122018
 
VMware Horizon View Icon

On VMware Horizon view after updating the view agent on the VM, you may notice that USB redirection stops working with the error “USB Redirection is not available for this desktop”. This is due to an issue with the certificates on the VDI host (The VM running the VDI OS), after the VMware view agent upgrade is completed.

To resolve this you must use MMC, open the local computer certificate store, browse to “VMwareView\Certificates”, delete the agent certificates (for the local agent), and finally reboot for the agent to regenerate the certificates.

See below for instructions:

  1. While connected to the VM running the VDI OS, click Start, type “mmc.exe” (without quotations), and open the Microsoft Management Console.
    mmc.exe

    Open MMC by running mmc.exe

     

  2. Open the “Add/Remote Snap-in” wizard.

    Open the Add/Remove Snap-in Wizard

     

  3. We must now open the local certificate store on the local computer. Select “Certicates” on the Available Snap-ins, click “Add”, select “Computer Account”, then proceed to choose “Local Computer” and complete the wizard.

    Select the Computer account certificate store on the local computer

     

  4. Expand the “Certificates (Local Computer)” on the left underneath “Console Root”. Expand “VMwareView”, then expand and select “Certificates”. Select the certificate on the right that matches the local computer name of the VDI host, right click and select “Delete”. You may have to do this multiple times if multiple certificates exist for the local computer.

    Delete the VMwareView local agent certificate

     

  5. Restart the VDI host. And USB redirection should now be working!

    VMware View USB Redirection issue resolved

     

Cheers to VDI!

May 172018
 
Digitally Accurate Inc. Logo

Looking for Calgary IT Managed Services or Calgary IT Consulting Services? We can help!

My company Digitally Accurate Inc. (https://www.digitallyaccurate.com/) has been helping businesses for almost 12 years with their IT strategies.

Feel free to reach out via E-mail, telephone, or LinkedIn to see how I can help! You can also visit the “Hire Stephen Wagner” tab, and yes, you’ll get to meet me! 🙂

My company is also partnered with numerous companies and can design, configure, and sell solutions including the following:

  • VMware Solution Design and Licensing
  • Veeam (Veeam Backup and Replication, Veeam Availability Suite, Veeam Backup Essentials)
  • HPe Servers, Storage, Networking
  • Aruba Networking
  • Microsoft Licensing (Including Office 365)
  • Sophos (Including Sophos UTM, and Sophos XG appliances)
  • 10ZiG Zero Clients
  • Duo Security (MFA)
  • Symantec (including Symantec Protection Suite)
  • Redhat (including RHEL: Redhat Enterprise Linux Server and Workstation)
  • Eaton UPS (Eaton Uninterrupted Power Supply and other Eaton power equipment)

Again, feel free to reach out for more information!

May 112018
 
Veeam Logo

This morning I noticed that CentOS 7.5 (1804) was available for upgrade via yum. After upgrading my CentOS instance from 7.4 to 7.5 on Microsoft Azure, I noticed that when running a backup using the Veeam Linux Agent, the system would crash and become completely unresponsive. I would have to manually restart the OS.

Upon reboot, I analyzed the console messages log and ran the backup again to see what was happening.

Here’s a look at my /var/log/messages:

May 11 07:24:46 HOSTNAME kernel: Request for unknown module key 'Veeam Software AG: 9d063645550b483bec752cb3c0249d5ede714b3e' err -11
May 11 07:24:46 HOSTNAME kernel: veeamsnap: loading out-of-tree module taints kernel.
May 11 07:24:46 HOSTNAME kernel: WARNING: module 'veeamsnap' built without retpoline-enabled compiler, may affect Spectre v2 mitigation
May 11 07:24:46 HOSTNAME kernel: veeamsnap: module verification failed: signature and/or required key missing - tainting kernel
May 11 07:24:46 HOSTNAME kernel: veeamsnap: applying kernel_stack fix up
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init Loading
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init Version: 2.0.0.400
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init Author: Veeam Software AG
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init licence: GPL
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init description: Veeam Snapshot Kernel Module
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init zerosnapdata: 1
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init debuglogging: 0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init snapstore enabled
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init start. container_alloc_counter=0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init start. container_sl_alloc_counter=0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init start. mem_cnt=0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init start. vmem_cnt=0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:ctrl_pipe_init .
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init Module major=243
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:blk_direct_bioset_create Specific bio set created.
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:blk_redirect_bioset_create Specific bio set created.
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:blk_deferred_bioset_create Specific bio set created.
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:snapimage_init .
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:snapimage_init Snapimage block device was registered. major=252
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init end. container_alloc_counter=0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init end. container_sl_alloc_counter=0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init end. mem_cnt=1
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init end. vmem_cnt=0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:ctrl_open file=0xffff95e4543b1800
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:ctrl_pipe_new .
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:ioctl_compatibility_flags Get compatibility flags
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:ioctl_tracking_collect Collecting tracking device:
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:tracking_collect Have not device under CBT.
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:tracking_add Adding. dev_id=8:1
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:tracker_Create dev_id 8:1
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:tracker_Create SectorStart    =0x800
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:tracker_Create SectorsCapacity=0xfa000
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:tracker_cbt_start .
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:cbt_map_create CBT map create.
May 11 07:24:47 HOSTNAME kernel: general protection fault: 0000 [#1] SMP
May 11 07:24:47 HOSTNAME kernel: Modules linked in: veeamsnap(OE) nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner xt_conntrack nf_conntrack iptable_security ext4 mbcache jbd2 dm_mirror dm_region_hash dm_log dm_mod sb_edac iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr i2c_piix4 sg hv_utils i2c_core ptp pps_core hv_balloon ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi ata_piix hv_storvsc hv_netvsc libata scsi_transport_fc hid_hyperv hyperv_keyboard scsi_tgt hyperv_fb crct10dif_pclmul crct10dif_common crc32c_intel hv_vmbus floppy serio_raw
May 11 07:24:47 HOSTNAME kernel: CPU: 1 PID: 1712 Comm: Lpb Server thre Tainted: G           OE  ------------   3.10.0-862.2.3.el7.x86_64 #1
May 11 07:24:47 HOSTNAME kernel: Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007  06/02/2017
May 11 07:24:47 HOSTNAME kernel: task: ffff95e447378000 ti: ffff95e45cbe0000 task.ti: ffff95e45cbe0000
May 11 07:24:47 HOSTNAME kernel: RIP: 0010:[]  [] page_array_memset+0x4d/0xa0 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: RSP: 0018:ffff95e45cbe3d60  EFLAGS: 00010246
May 11 07:24:47 HOSTNAME kernel: RAX: 0000000000000000 RBX: ffff95e449615200 RCX: 0000000000000200
May 11 07:24:47 HOSTNAME kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff187288716000
May 11 07:24:47 HOSTNAME kernel: RBP: ffff95e45cbe3d60 R08: ffffffffbe274fef R09: ffff95e460affa60
May 11 07:24:47 HOSTNAME kernel: R10: ffff95e460affa60 R11: 0000000000000000 R12: 0000000000000001
May 11 07:24:47 HOSTNAME kernel: R13: 00000000000fa000 R14: 0000000000000000 R15: 0000000000800001
May 11 07:24:47 HOSTNAME kernel: FS:  00007f336f7fe700(0000) GS:ffff95e495640000(0000) knlGS:0000000000000000
May 11 07:24:47 HOSTNAME kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 11 07:24:47 HOSTNAME kernel: CR2: 0000000000738df0 CR3: 00000002d3afc000 CR4: 00000000001406e0
May 11 07:24:47 HOSTNAME kernel: Call Trace:
May 11 07:24:47 HOSTNAME kernel: [] cbt_map_allocate+0x6e/0x160 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: [] cbt_map_create+0x73/0x100 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: [] tracker_cbt_start+0x5a/0xc0 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: [] tracker_Create+0x16a/0x650 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: [] tracking_add+0x2e0/0x450 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: [] ioctl_tracking_add+0x6c/0x170 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: [] ctrl_unlocked_ioctl+0x4e/0x60 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: [] do_vfs_ioctl+0x350/0x560
May 11 07:24:47 HOSTNAME kernel: [] ? __sb_end_write+0x31/0x60
May 11 07:24:47 HOSTNAME kernel: [] ? vfs_write+0x182/0x1f0
May 11 07:24:47 HOSTNAME kernel: [] SyS_ioctl+0xa1/0xc0
May 11 07:24:47 HOSTNAME kernel: [] system_call_fastpath+0x1c/0x21
May 11 07:24:47 HOSTNAME kernel: Code: 01 49 89 f9 48 0f af c2 49 8b 79 10 ba 00 10 00 00 40 f6 c7 01 75 47 40 f6 c7 02 75 51 40 f6 c7 04 75 2b 89 d1 c1 e9 03 83 e2 07  48 ab 74 0e 41 89 c8 83 c1 01 39 d1 42 88 34 07 72 f2 49 83
May 11 07:24:47 HOSTNAME kernel: RIP  [] page_array_memset+0x4d/0xa0 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: RSP 
May 11 07:24:47 HOSTNAME kernel: ---[ end trace 96b51a664f4baef9 ]---

It appeared the veeamsnap module was causing a protection fault with the kernel, causing the system to crash.

In an effort to troubleshoot, I uninstalled and reinstalled Veeam and tried rebuilding the kernel module, however the issue still persisted. After some searching, I came across these two posts:

https://forums.veeam.com/veeam-agent-for-linux-f41/veeam-agent-for-linux-and-rhel-7-5-crash-t50170.html

https://www.veeam.com/kb2569

According to the KB, the veeamsnap module isn’t compatible with kernel version 3.10.0-862.

Checking my system after upgrading CentOS 7.5:

[root@HOSTNAME ~]# uname -a
Linux HOSTNAME.somedomain.com 3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9 18:05:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@HOSTNAME ~]# cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)

 

Essentially, as of right now the Veeam Agent for Linux is not yet supported on CentOS 7.5, RHEL 7.5, or Oracle RHCK 7.5. If you’re running any of these, hold off and do not install Veeam Agent for Linux. If you are scheduling an upgrade, do not perform upgrade unless you want to break your backup. It sounds like this should be fixed in a future update.

May 082018
 

Recently a customer of mine who is using an outdated version of Intuit QuickBooks on their Terminal Server (RDS Remote Desktop Services) started to experience an issue when users attempted to log on. QuickBooks would initialize, then prompt the TLS 1.2 warning message, and then completely crash. This would prevent the users from doing any tasks.

In an effort to troubleshoot this, I tried to use different accounts, checked the QBW.ini file for any flags that could dismiss this message, tried Intuit’s “TLS preparedness tool” (which still scares me because I have no idea what system changes it made on the server), etc… All of these attempts had no effect on the issue.

For a temporary workaround, you’ll need to load up QuickBooks on a workstation (not using terminal services). You’ll need to open the datafile, select “Do Not Show this again”, and then close. You’ll need to do this for each datafile. Please note that if you do not receive the prompts on other datafiles, you’ll need to open the datafile with a different Quickbooks username (QB account, not Windows account) in order to get it to prompt.

This issue should only occur on older versions of Quickbooks when using TS/RDS. To officially resolve this issue, I recommend upgrading to a more recent (and in support) version of Quickbooks.

May 062018
 
DUO

I’m a big fan of MFA, specifically Duo Security‘s product (I did a corporate blog post here). I’ve been using this product for some time and use it for an extra level of protection on my workstations, servers, and customer sites. I liked it so much so that my company (Digitally Accurate Inc.) became a partner and now resells the services.

Today I want to write about a couple issues I had when deploying the pam_duo module on CentOS Linux 7. The original duo guide can be found at https://duo.com/docs/duounix, however while it did work for the most part, I noticed there were some issues with the pam configuration files, especially if you are wanting to use Duo MFA with usernames and passwords, and not keys for authentication.

A symptom of the issue: I noticed that when following the instructions on the website for deployment, after entering the username, it would skip the password prompt, and go right for DUO authentication, completely bypassing the password all together. I’m assuming this is because the guide was written for key authentication, but I figured I’d do a quick crash-course post on the topic and create a simple guide. I also noticed that sometimes even if an incorrect password was typed in, it would allow authentication if DUO passed as successful.

Ultimately I decided to learn about PAM, understand what it was doing, and finally configure it properly. Using the guide below I can confirm the password and MFA authentication operate correctly.

To configure Duo MFA on CentOS 7 for use with usernames and passwords

First and foremost, you must log in to your Duo Account and go to applications, click “Protect an Application” and select “Unix Application”. Configure the application and document/log your ikey, secret key, and API hostname.

Now we want to create a yum repo where we can install, and keep the pam_duo module up to date. Create the file /etc/yum.repos.d/duosec.repo and then populate it with the following:

[duosecurity]
name=Duo Security Repository
baseurl=http://pkg.duosecurity.com/CentOS/$releasever/$basearch
enabled=1
gpgcheck=1

We’ll need to install the signging key that the repo uses, and then install the duo_unix package. By using yum, we’ll be able to keep this package regularly up to date when we update the server. Run the following commands:

rpm --import https://duo.com/RPM-GPG-KEY-DUO
yum install duo_unix

Configure the pam_duo module by editing the /etc/duo/pam_duo.conf file. You’ll need to populate the lines with your ikey, secret key, and API hostname that you documented above. We use “failmode=safe” so that in the event of an internet disconnection, we can still login to the server without duo. It’s safe to enable this fail-safe, as the purpose is to protect it against the internet. Please see below:

[duo]
; Duo integration key
ikey = XXXXXXXXXXXXXXXXXXXX
; Duo secret key
skey = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
; Duo API host
host = XXXXXXXXX.duosecurity.com
; Send command for Duo Push authentication
pushinfo = yes
; failmode safe if no internet it works (secure locks it up)
failmode = safe

Configure sshd to allow Challenge Response Authentication by editing /etc/ssh/sshd_config, then locate and change “ChallengeResponseAuthentication” to yes. Please note that the line should already be there, and you should simply have to move the comment symbol to comment the old line, and uncomment the below line as shown below:

ChallengeResponseAuthentication yes

And now it gets tricky… We need to edit the pam authentication files to incorporate the Duo MFA service in to it’s authentication process. I highly recommend that throughout this, you open (and leave open) an additional SSH session, so that if you make a change in error and lock yourself out, you can use the extra SSH session to revert any changes to the system to re-allow access. It’s always best to make a backup and copy of these files so you can easily revert if needed.

DISCLAIMER: I am not responsable if you lock yourself out of your system. Please make sure that you have an extra SSH session open so that you can revert changes. It is assumed you are aware of the seriousness of the changes you are making and that you are taking all precautions (including a backup) to protect yourself from any errors.

Essentially two files are used for authentication that we need to modify. One file is for SSH logins, and the other is for console logins. In my case, I wanted to protect both methods. You can do either, or both. If you are doing both, it may be a good idea to test with SSH, before making modifications to your console login, to make sure your settings are correct. Please see below for the modifications to enable pam_duo:

/etc/pam.d/password-auth (this file is used for SSH authentication)

#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth        required      pam_env.so
auth        required      pam_faildelay.so delay=2000000
#auth        sufficient    pam_unix.so nullok try_first_pass
auth        requisite     pam_unix.so nullok try_first_pass
auth  sufficient pam_duo.so
auth        requisite     pam_succeed_if.so uid >= 1000 quiet_success
auth        required      pam_deny.so

account     required      pam_unix.so
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 1000 quiet
account     required      pam_permit.so

password    requisite     pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type=
password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok


password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
-session     optional      pam_systemd.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so

/etc/pam.d/system-auth (this file is used for console authentication)

auth        required      pam_env.so
auth        sufficient    pam_fprintd.so
#auth        sufficient    pam_unix.so nullok try_first_pass
# Next two lines are for DUO mod
auth        requisite     pam_unix.so nullok try_first_pass
auth        sufficient    pam_duo.so
auth        requisite     pam_succeed_if.so uid >= 1000 quiet_success
auth        required      pam_deny.so

account     required      pam_unix.so
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 1000 quiet
account     required      pam_permit.so

password    requisite     pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type= ucredit=-1 lcredit=-1 dcredit=-1 ocredit=-1
password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok remember=5
password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
-session     optional      pam_systemd.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so

Now, we must restart sshd for the changes to take affect. Please make sure you have your extra SSH session open in the event you need to rollback your /etc/pam.d/ files. Restart the sshd service using the following command:

service sshd restart

Attempt to open a new SSH session to your server. It should now ask for a username, password, and then prompt for Duo authentication. And you’re done!

More information on Duo Multi Factor Authentication (MFA) can be found here.

May 062018
 
Sophos XG and Sophos UTM

Today I’m going to be talking about connecting a Sophos XG firewall to a Sophos UTM firewall for a site-to-site VPN connection specifically using SSL tunneling. Furthermore we are doing this to connect a Microsoft Azure Virtual Network (using a Sophos XG instance) to an On-Premise LAN (running a Sophos UTM).

This type of connection and configuration is standard for corporations, businesses, and organizations who have workloads on Microsoft Azure and need to connect their Azure environment to their corporate LAN. To learn how to deploy Sophos XG in Microsoft Azure, please read this post.

WARNING AND DISCLAIMER: Following the steps in this document if done incorrectly or if your environment is different from the one used in this example, can cause network connectivity issues or a loss of connectivity. An assumption is made that you are skillful enough to know what tasks you are performing and what result they may have on your own environment. You may need to revert these steps if connectivity is lost to restore access to your environment.

Let’s ask some key questions and get some answers:

  • Why are we using both (2) products, Sophos XG and Sophos UTM for the connection instead of using the same product on each end?
    • Sophos only supports deployment of the Sophos XG on Microsoft Azure. Sophos UTM cannot be deployed on Azure. Numerous companies and organizations are still using the Sophos UTM product for their on-premise IT infrastructure. There is a need to have these different products co-exist and function together, like in this specific example.
  • Why are we using a Sophos XG Appliance/Instance to handle the VPN on Microsoft Azure, instead of using the Microsoft branded Azure VPN Gateway?
    • Microsoft Azure has a VPN gateway appliance which can handle the Azure side of the VPN connection, however this is a resource that costs money (instance time and instance data transfer) and has limited configuration options. Numerous companies and organizations are using Sophos XG instances on Azure to protect their internet facing workloads already. Instead of paying for 2 resources (Sophos XG and Microsoft VPN Gateway), you can consolidate and use one for both. You also have extra functionality and security options when using the Sophos XG instance to handle VPN traffic (IPS, strict firewall rules, packet inspection, etc). The Sophos firewalls can be centrally managed/monitored via Sophos Management Products (Sophos SUM (Sophos UTM Manager), and/or Sophos CFM (Sophos Central Firewall Manager)).
  • Why are we using an SSL VPN connection instead of IPSec or other type of VPN?
    • Microsoft Azure blocks some IP Protocol traffic within Virtual Networks. As quoted: “IP-in-IP encapsulated packets, and Generic Routing Encapsulation (GRE) packets are blocked within VNets” (per https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-faq), which means that while you could configure an IPsec tunnel, none of the traffic would pass through the Virtual Networks. We utilize SSL VPNs to overcome this limitation as all traffic goes over the SSL connection.

Let’s address what components we will discuss in this post:

The information inside of this post can be used for any of the 4 components above and don’t necessarily have to be used in the same configuration. An example: This guide could be used by someone wanting to connect an on-premise XG and on-premise UTM unit together via SSL VPN (with no need or use for Azure). Another example: The section on routing tables can provide information for someone using a different network security product on Microsoft Azure. However, the ultimate goal of this article is to address all four of the components together for a complete end to end deployment.

Now let’s get to the configuration of each of the four components.

Deploy a Sophos XG instance/appliance to Microsoft Azure

In a previous post, I covered how to deploy a new Sophos XG firewall appliance/instance to Microsoft Azure, specifially allowing deployment to existing resource groups. The full URL (and instructions) can be found at https://www.stephenwagner.com/2018/05/05/deploy-sophos-xg-firewall-microsoft-azure-existing-resource-group/

Sophos XG to Sophos UTM SSL VPN Connection Configuration and Encryption Settings

We will configure the SSL VPN settings on both the Microsoft Azure Sophos XG appliance/instance, and the on-premise Sophos UTM appliance/instance. Afterwards, we will create a tunnel, configure it, enable it, and establish connectivity between the two Sophos firewall instances. During this process, we’ll be configring the SSL VPN settings on both appliance/instances, configuring the tunnel, configuring encryption settings, and establishing VPN communication.

Please note, with this configuration there are essentially 3 networks: the Azure network, the SSL transit network (this is the network in between the networks we are connecting that is part of the SSL VPN), and the on-premise network. When you configure your firewall rules (not in the scope of this document), you must factor this in and allow applicable traffic to/from the SSL network so that the packets can pass. This SSL transit network is specified in the “Show VPN Settings” on the XG unit under “SSL VPN”, and “IPv4 Lease Range”. This network must be different and not overlap any subnets you are using on both your Azure network, and on-premise network. In my case I chose something way off in an entirely different IP space (172.16.0.0), and I suggest you do as well.

Follow these steps to configure the SSL VPN settings on both Sophos XG and UTM appliances/instances.

  1. Log on to the Sophos XG appliance, select “VPN” under “Configure” on the left hand side, and then select the gear icon with “Show VPN Settings” on the top right. See example.

    Sophos XG Show VPN Settings

    Sophos XG Show VPN Settings

  2. Configure your “SSL VPN Settings” and “Cryptographic Settings” to be similar to the example below. Please modify to reflect your own environment. Cryptographic settings should match example below. Please note that the “IPv4 Lease Range” is not for your Azure or Internal LAN subnet, but actually the subnet used by the SSL VPN server. This value has to be an entirely new subnet dedicated for SSL VPN functions.

    Sophos XG SSL VPN Settings

    Sophos XG SSL VPN Settings

  3. Log on to the Sophos UTM appliance, select “Site-to-Site VPN”, then select “SSL”, then click on the “Advanced” tab.
  4. Configure your “Advanced” settings to reflect the following below.

    Sophos UTM SSL Site-to-Site VPN Advanced Settings

    Sophos UTM SSL Site-to-Site VPN Advanced Settings

You have now configured the general SSL VPN Advanced settings, we can now move on to configuring the tunnel itself.

To configure the SSL Site to Site VPN tunnel between the Sophos appliances, we’ll need to configure the Sophos XG (on Azure) to act as a server, and the Sophos UTM (on prem) which will act as the client. Side note: In my own testing, I found that the XG had to be the server in order to get them to connect.

To configure the SSL VPN tunnel Server on the Sophos XG:

  1. Log on to your Sophos XG interface, click on “VPN” under “Configure” on the left hand side, and then choose “SSL VPN (Site-to-Site)” from the top. Then click “Add” under the “Server” section. As shown below.

    Sophos XG SSL VPN (Site-to-Site) Settings

    Sophos XG SSL VPN (Site-to-Site) Settings

  2. Now give the VPN connection a name, enter a friendly description, and specify Local Networks (Azure Subnets), and Remote Networks (On-Prem Networks). And then click “Save” as shown below.

    Sophos XG Create new SSL VPN Server

    Sophos XG Create new SSL VPN Server

  3. Now we need to download the configuration so that we can load it on to your Sophos UTM so it know’s how to connect. While still in the “SSL VPN (Site-to-Site)” window, look for the column called “Manage”, and click the download icon (you can confirm it’s download via the mouse-over description). As shown below.

    Sophos XG SSL VPN Server Download Config

    Sophos XG SSL VPN Server Download Config

  4. Save this file to your computer as we’ll need it for configuring the Sophos UTM

To configure the SSL VPN tunnel Client on the Sophos UTM:

  1. Log on to your Sophos UTM web interface, click on “Site-to-Site VPN” on the left hand side, and then select “New SSL Connection”. As shown below in step 2.
  2. Set the Connection type to “Client”, give it a friendly name. Now click on the folder next to “Configuration File” to brows to the VPN config file (this is the file we downloaded above from the Sophos XG unit). In my case, I also selected the “Override peer hostname” as I wanted to over the hostname of the Sophos XG unit (to avoid problems I chose an IP address). This value sets the hostname of the server that the UTM is connecting to. We also uncheck the “Automatic firewall rules” as we don’t want any rules automatically created, we want to specify only what is needed.

    Sophos UTM Create New SSL VPN Connection

    Sophos UTM Create New SSL VPN Connection

  3. Hit Save, and you should be left with something like below.

You have now fully configured the SSL Site-to-Site VPN tunnel between your Sophos XG and Sophos UTM instances.

To confirm a functioning VPN tunnel on your Sophos XG unit, you should see something similar to below.

Sophos XG SSL VPN (Site-to-Site) Active VPN Status

Sophos XG SSL VPN (Site-to-Site) Active VPN Status

To confirm a functioning VPN tunnel on your Sophos UTM unit, you should see something simialar to below.

Sophos UTM Active VPN Status

Sophos UTM Active VPN Status

Please note that a bug with XG to UTM VPN, is that on the Sophos UTM reports the active subnets as “unknown” as shown above on both sides. This can be safely ignored.

You can start to test some basic communication, however you still need to create firewall rules. Please note that the Azure network will not be routable until you continue the steps below and configure proper routing.

Microsoft Azure VNet (Virtual Network) custom routing table

When you create a Virtual Network (VNet) in Microsoft Azure, Azure will handle the routing for you automatically. It will create routers and other “instances” to handle network connectivity as you provision new subnets, gateways, devices, and network paths. Since we are deploying our own router, we want to override these default routing tables that are automatically created.

When looking at our target configuration, our Sophos XG unit will have an internet facing static IP, and will be handling communication between our internal network (and hosts), the internet (outside world), and our internal on-premise network (LAN). Because of this, we have to make changes to our Azure enviroinment so that the default subnet network route becomes the Internal IP Address of the Sophos XG firewall. We need to configure routes for both our Azure subnets (if wanted), our corporate on-premise LAN, and our catch-all route for internet access (0.0.0.0/0).

Once we create our own routing table, we’ll need to assign it to specific subnets to make those specific subnets enforce the routes.

To create a custom Route Table:

  1. Log in to the Azure Portal, View “All Resources”, click on the “Add” button at the top to create a new resource, or simply click on “Create a resource” on the top left of the Azure Portal.
  2. Select Networking on the left side of the table, then select “Route Table”. See example below.

    Sophos XG Azure Add Resource Route Table

    Sophos XG Azure Add Resource Route Table

  3. Populate the fields, select the subscription, resource group, and the Location of where you’d like to place this. For “Resource Group”, select “Use existing”, and then specify the Resource Group you are attaching this to.

    Sophos XG Azure Create Route Table

    Sophos XG Azure Create Route Table

  4. Select “Create” to complete the creation of the route table.
  5. Now that we have a customer routing table, we want to create a route. With the “Route Table” object still open, open the “Routes” tab to open the page from the column on the left.
  6. Click on Add, and then create a route for the CIDR block that covers both your Azure subnets, and corporate on-premise LAN subnet. PLEASE NOTE: This CIDR block (Address prefix) should be large enough that it includes both your Azure subnets and your on-premise network. This will make this rule apply to all traffic destined for those networks. Under “Next Hop Type”, select “Virtual Appliance”, and then enter the IP address of your Sophos XG LAN Network interface. Then click save.

    Sophos XG Azure Route Table LAN Route

    Sophos XG Azure Route Table LAN Route

  7. Click on Add, and then create a catch all route for the internet/WAN with an address prefix of 0.0.0.0/0. Under “Next Hop Type”, select “Virtual Appliance”, and then enter the IP address of your Sophos XG LAN Network interface. Then click save.

    Sophos XG Azure Route Table Internet Route

    Sophos XG Azure Route Table Internet Route

  8. The Route’s section should look similar to this.

    Sophos XG Azure Route Table Route List

    Sophos XG Azure Route Table Route List

  9. Now we need to assign the route table to the chosen subnets. This will apply the routes we created above to specific subnets on our Azure Virtual Network (VNet). With the “Route Table” object still open, open the “Subnets” page from the column on the left.
  10. Click on “Associate” to add this to an existing subnet. And then associate it to your Azure Virtual Network subnet where the XG and your VMs reside.

    Sophos XG Azure Route Table Associate Subnet

    Sophos XG Azure Route Table Associate Subnet

  11. Now completed, the “Overview” tab should look similar to this.

    Sophos XG Azure Route Table Overview

    Sophos XG Azure Route Table Overview

You’ve successful configured a custom routing table for your Microsoft Azure subnet which will route packets destined for other subnets (including internet/WAN) to your Sophos XG for further routing.

Microsoft Azure enable IP Forwarding on Virtual Network Interface

In order for a VM (Virtual Machine) to have the ability to forward and route packets, we need to enable “IP Forwarding” on both the Internal and External NIC of the Sophos XG appliance running on Azure.

To enable this:

  1. Log in to the Azure Portal, view “All Resources” and select one of the “Network interface” objects for your Sophos XG appliance/instance.
  2. Click on “IP configurations” under the “SETTINGS” group.
  3. Look for “IP forwarding” under “IP forwarding settings”. Set this toggle to “Enabled” as per below.

    Enable IP Forwarding on Microsoft Azure Network Interface

    Enable IP Forwarding on Microsoft Azure Network Interface

  4. Click Save
  5. Repeat for the other Sophos Network interface (both External and Internal need this enabled)

IP Forwarding has now been enabled. The Sophos XG appliance can now successfully route packets in your Microsoft Azure Virtual Network (VNet).

Conclusion

At this point you’ll have everything configured. You’ll have a SSL VPN between your Azure hosted Sophos XG instance and your on-premise Sophos UTM, as well as connectvity between both of the networks. You will now need to configure all your firewall rules to only permit the traffic you want to traverse from internal-azure to internal-onprem as well as external WAN traffic (this is beyond the scope of this document). You need to take care in making sure you only permit traffic that should be going over these network links. Now that both networks are connected, it provides another means to connect and communicate with the other networks which increases your security risks. You’re not only securing against one internet connection on one LAN, but 2 internet connections across 2 networks.

In my scenario by configuring this I was able to decommission the Microsoft Azure VPN Gateway (minimizing costs), and have my own security appliance/instance handle the communication between both networks and also protect both networks with all the fancy features that the Sophos XG and UTM line offer.

Leave a comment with feedback!