Aug 202018
 

An all too common problem is when users report e-mail delays ranging from 5 to 15 minutes. When troubleshing these types of issues, you’ll notice this commonly occurs when receiving e-mails from organizations that use Office 365. Specifically this occurs due to greylisting.

Why does this happen

You’re organization is using greylisting on your e-mail proxy/SMTP relay to reduce spam. Greylisting temporarily rejects the first send of an e-mail and waits for the sending server to re-transmit the message. This process usually takes around 5-15 minutes to complete. Greylisting is used because spammers won’t re-transmit the message, which leads to a massive reduction of spam messages coming through.

Once the sending server retransmits, the sending server IP address is added to your firewalls “safe senders” whitelist. From this point on the IP address (or server) will not be subject to greylisting (and any subsequent e-mails).

Office 365 has hundreds, if not thousands (possibly 10’s of thousands) of servers they use to transmit e-mail. The chance of multiple e-mails being sent from a single server is very slim, therefor greylisting is applied to every IP (server) that is sending e-mail because it’s different. Each e-mail from an Office 365 user can take 5-15 minutes, since a new server is used every time.

How to resolve

You’ll need to configure and add an exception to your e-mail proxy/SMTP relay/firewall. This exception can be based off domain, DNS name of sending server, or IP address ranges.

Scroll down for instructions on how to create an exception on a Sophos UTM.

Domain Exception

If you use domain based exceptions, you’ll need to configure these manually for each sending domain that you want your firewall to skip greylist checking. This is a very manual process, which requires lots of human intervention to continuously update your greylist exception.

DNS FQDN of MX Server

This method is the easiest, however most firewall or UTM’s will now allow these types of exceptions since a number of DNS queries will be needed everytime an e-mail comes in. One DNS query on the MX record, and then another DNS query on the DNS host contained in the MX record. If you can configure this type of exception, you’ll want to configure it as below:

*-com.mail.protection.outlook.com

IP Address Range

This is the best method. To create an IP address range exception, we’ll need a copy of all the IP address ranges or IP address spaces that Office 365 uses to send mail. This list can be found at: https://docs.microsoft.com/en-us/office365/SecurityCompliance/eop/exchange-online-protection-ip-addresses.

We’ll need to create an exception that skips greylist checking on the IP addresses outlined in the above link. This will stop any greylist checking on e-mails from Office 365 servers.

In my case, I use a Sophos UTM firewall, and to create an exception I had to do the following:

  1. Log on to the Webmin interface.
  2. Select “Email Protection”, then “STMP” on the left hand side, then “Exceptions” tab at the top.

    Sophos UTM E-Mail and SMTP Exception List

    Sophos UTM E-Mail and SMTP Exception List

  3. Create a “New Exception List” and call it “Office 365 GreylistWhitelist”.
  4. Check the “Greylisting” box under “Antispam”, and then check the “For these source hosts/networks”.

    Sophos UTM SMTP Create Exception

    Sophos UTM SMTP Create Exception

  5. Click the “+” button, and call the Network Definition “Exchange365-EOP-Group”. Change the type to “Network Group”.
  6. Click the “+” button in the members section, and start adding the IP spaces. Repeat this for each IP space (in total I added 23). Each network name (IP address space) requires a unique name, I named mine “Exchange365-EOP1” through “Exchange365-EOP23”.

    Sophos UTM SMTP Configure Exception

    Sophos UTM SMTP Configure Exception

  7. Click Save on the Network Group, and click Save on the exception.
  8. Enable the Exception

    Sophos UTM SMTP Exception Rule

    Sophos UTM SMTP Exception Rule

  9. Completed! You’ve now made the exception and delays should no longer occur.
Aug 192018
 

I finally got around to mounting my Wilson weBoost Home 4G Cell Phone Booster Antenna on the roof. Here’s some pictures of the completed install. I’ve had this booster for a while and it’s worked great, however some new cell towers went up in the area, and I wanted to stop using the window mount and re aim the antenna.

Wilson weBoost Home 4G Cell Phone Booster Roof Outdoor Antenna

Wilson weBoost Home 4G Cell Phone Booster Roof Outdoor Antenna

For those of you wanting to read my original post on the Wilson weBoost Home 4G Cell Phone Booster Kit, installation, and a review, you can find it at https://www.stephenwagner.com/2017/06/01/cellmobile-phone-reception-issues-resolve-with-a-wilson-amplifier-cell-booster/.

The house that I live in, actually had a roof mounted satellite dish that was no longer in use (used before the provider ran coax in the area). The dish, roof mount, and coax were all in place, however the coax was cut so I couldn’t re-use it.

I was able to remove 2 of the bolts on the satelite dish to remove it from the pole mount, and proceeded to install the antenna on the pole using the outdoor mounting kit included with the cell booster. I was extremely pleased with the install.

See below for more pics:

Roof mounted Wilson weBoost Home 4G Cell Phone Booster Kit

Roof mounted Wilson weBoost Home 4G Cell Phone Booster Kit

Roof mounted Wilson weBoost Home 4G Cell Phone Booster Kit Cabling

Roof mounted Wilson weBoost Home 4G Cell Phone Booster Kit Cabling

Roof mounted Antenna pole mount

Roof mounted Antenna pole mount

The cabling goes through the pole, down to the eavestrough where I have it zip-stripped (yet elevated) along the roof until I get to the house’s siding. I was able to tuck it in the corner siding down to the wiring access panel for the house, then into the house through the hole.

After mounting it, it took around 30 minutes to aim it with the assistance of the “LTE Discover” Android app (available at https://play.google.com/store/apps/details?id=net.simplyadvanced.ltediscovery). Remember, when aiming your antenna, it’s important to unplug your booster for 5-10 seconds for it to fully reset for it to function with the new antenna position.

Again, make sure you check out my original post and review at https://www.stephenwagner.com/2017/06/01/cellmobile-phone-reception-issues-resolve-with-a-wilson-amplifier-cell-booster/!

Aug 182018
 
CentOS Logo

Let’s say that you’re hosting someone’s equipment and they start to abuse their connection speed. Let’s say that you’re limited in your bandwidth, and you want to control your own bandwidth to make sure you don’t max out your own internet connection. You can take care of both of these problems by building your own traffic shaping network control device using CentOS and using the “tc” linux command.

In this post I’m going to explain what traffic shaping is, why you’d want to use traffic shaping, and how to build a very basic traffic shaping device to control bandwidth on your network.

What is traffic shaping

Traffic shaping is when one attempts to control a connection in their network to prioritize, control, or shape traffic. This can be used to control either bandwidth or packets. In this example we are using it to control bandwidth such as upload and download speeds.

Why traffic shaping

For service providers, when hosting customer’s equipment, the customer may abuse their connection or even max it out legitimately. This can put a halt on the internet connection if you share it with them, or cause bigger issues if it’s shared with other customers. In this example, you would want to implement traffic shaping to allot only a certain amount of bandwidth so they wouldn’t bring the internet connection or network to a halt.

For normal people (or a single business), as fast as the internet is today, it’s still very easy to max your connection out. When this happens you can experience packet loss, slow speeds, and interruption of services. If you host your own servers this can cause even a bigger issue with interruption of those services as well. You may want to limit your own bandwidth to make sure that you don’t bring your internet to a halt, and save some for other devices and/or users.

Another reason is just to implement basic QoS (Quality of Service) across your network, to keep usage and services in harmony and eliminate any from hogging the network connections up.

How to build your own basic traffic shaping device with CentOS and tc

In this post we will build a very simple traffic shaping device that limits and throttles an internet connection to a defined upload and download speed that we set.

You can do this with a computer with multiple NICs (preferably one NIC for management, one NIC for internet, and one NIC for network and/or the hosts to be throttled). If you want to get creative, there are also a number of physical network/firewall appliances that are x86 based, that you can install Linux on. These are very handy as they come with many NICs.

When I set this up, I used an old decommissioned Sophos UTM 220 that I’ve had sitting around doing nothing for a couple years (pic below). The UTM 220 provides 8 NICs, and is very easy to install Linux on to.

Sophos UTM 220 Running CentOS Linux

Sophos UTM 220 Running CentOS Linux

Please Note: The Sophos UTM 220 is just a fancy computer in a 1U rack mounted case with 8 NICs. All I did was install CentOS on it like a normal computer.

Essentially, all we’ll be doing is installing CentOS Linux, installing “tc”, configuring the network adapters, and then configuring a startup script. In my example my ISP provides me 174Mbps download, and 15Mbps upload. My target is to throttle the connection to 70Mbps download, and 8Mbps upload. I will allow the connection to burst to 80Mbps down, and 10Mbps up.

To get started:

  1. Install CentOS on the computer or device. The specifics of this are beyond the scope of this document, however you’ll want to perform a minimal install. This device is strictly acting as a network device, so no packages are required other than the minimal install option.
  2. During the CentOS install, only configure your main management NIC. This is the NIC you will use to SSH to, control the device, and update the device. No other traffic will pass through this NIC.
  3. After the install is complete, run the following command to enable ssh on boot:
    chkconfig sshd on
  4. Install “tc” by running the command:
    yum install tc
  5. Next, we’ll need to locate the NIC startup scripts for the 2 adapters that will perform the traffic shaping. These adapters are the internet NIC, and the NIC for the throttled network/hosts. Below is an example of one of the network startup scripts. You’re NIC device names will probably be different.
    /etc/sysconfig/network-scripts/ifcfg-enp2s0
  6. Now you’ll need to open the file using your favorite text editor and locate and set ONBOOT to no as shown below. You can ignore all the other variables. You’ll need to repeat this for the 2nd NIC as well.
    TYPE=Ethernet
    PROXY_METHOD=none
    BROWSER_ONLY=no
    BOOTPROTO=dhcp
    DEFROUTE=yes
    IPV4_FAILURE_FATAL=no
    IPV6INIT=yes
    IPV6_AUTOCONF=yes
    IPV6_DEFROUTE=yes
    IPV6_FAILURE_FATAL=no
    IPV6_ADDR_GEN_MODE=stable-privacy
    NAME=enp2s0
    UUID=xxxxxxxx-xxxx-xxx-xxxx-xxxxxxxxxxxx
    DEVICE=enp2s0
    ONBOOT=no
  7. Now we can configure the linux startup script to configure a network bridge between the two NICs above, and then configure the traffic shaping rules with tc. Locate and open the following file for editing:
    /etc/rc.d/rc.local
  8. Append the following text to the rc.local file:
    # Lets make that bridge
    brctl addbr bridge0
    
    # Lets add those NICs to the bridge
    brctl addif bridge0 enp5s0
    brctl addif bridge0 enp2s0
    
    # Confirm no IP set to NICs that are shaping
    ifconfig enp5s0 0.0.0.0
    ifconfig enp2s0 0.0.0.0
    
    # Bring the bridge online
    ifconfig bridge0 up
    
    # Clear out any existing tc policies
    tc qdisc del dev enp2s0 root
    tc qdisc del dev enp5s0 root
    
    # Configure new traffic shaping policies on the NICs
    # Set the upload to 8Mbps and burstable to 10mbps
    tc qdisc add dev enp2s0 root tbf rate 8mbit burst 10mbit latency 50ms
    # Set the download to 70Mbps and burstable to 80Mbps
    tc qdisc add dev enp5s0 root tbf rate 70mbit burst 80mbit latency 50ms
    
  9. Restart the linux box:
    shutdown -r now
  10. You now have a traffic shaping network device!

Final Thoughts

Please note that normally you would not place the script in the rc.local file, however we wanted something quick and simple. The script may not survive in the rc.local file when updates/upgrades are applied against on the Linux install, so keep this in mind. You’ll also need to test to make sure that you are throttling in the correct direction with the 2 NICs. Make sure you test this setup and allow time to confirm it’s working before putting it in a production network.

Aug 182018
 
VMware vSphere Mobile Watchlist Logo

Did you know that you can monitor and manage your VMware vSphere environment (ESXi hosts, cluster, and VMs) remotely with the “VMware vSphere Mobile Watchlist” app on your Android phone? Well, you can!

Download link: https://play.google.com/store/apps/details?id=com.vmware.beacon

The VMware vSphere Mobile Watchlist (VMware Watchlist) Android App

For some time now, I’ve been using this neat little app from VMware (available for download here) to monitor and manage my vSphere cluster remotely. You can use the app while directly on your LAN, or via VPN (I use it with OpenVPN to connect to my Sophos UTM). I’ve even used it while on airplanes using the on board in-flight WiFi.

The reason why I’m posting about this, is because I’ve never actually heard anyone talk about the app (which I find strange), so I’m assuming others aren’t aware of it’s existence as well.

The app runs extremely well on my Samsung S8+, Samsung S9+, and Samsung Tab E LTE tablet. I haven’t run in to any issues or app crashes.

Let’s take a look at the app

vSphere Mobile Watchlist Login Prompt

vSphere Mobile Watchlist Login Prompt

The above screen is where you initially log in. I use my Active Directory credentials (since I have my vCenter server integrated with AD).

vSphere Mobile Watchlist Hosts and VM list

vSphere Mobile Watchlist Hosts and VMs

In the default view (shown above), you can view a brief summary of your ESXi hosts, as well as a list of virtual machines running.

vSphere Mobile Watchlist Host Information

vSphere Mobile Watchlist Host Information

After selecting an ESXi host, you can view the hosts resources, details, related objects, as well as flip over to view host options.

vSphere Mobile Watchlist Host Options

vSphere Mobile Watchlist Host Options

Under host options, you can Enter Maintenance mode, reboot the host, shutdown the host, disconnect the host, or view the hosts’ sensor data.

vSphere Mobile Watchlist Host Sensor List and Fan Data

vSphere Mobile Watchlist Host Sensor Data (Fans)

Checking the HPe Proliant DL360p Gen8 fan sensor data with VMware Watchlist.

vSphere Mobile Watchlist Host Sensor Data (Temperature Sensor List)

vSphere Mobile Watchlist Host Sensor Data (Temperature)

Checking the HPe Proliant DL360p Gen8 temperature sensor data with VMware Watchlist. While not shown above, you can select individual items to pull the actual temperature values. Please Note that the temperature values are missing a decimal (Example: 2100 = 21.00 Celsius).

vSphere Mobile Watchlist VM Information

vSphere Mobile Watchlist VM Information View

When selecting a VM (Virtual Machine) from the default view, you can view the VM’s Resources (CPU, Memory, and Storage), Details (IP Addresses, DNS hostnames, Guest OS, VMWare Tools Status), related objects, and a list of other VMs running on the same host.

vSphere Mobile Watchlist VM Options

vSphere Mobile Watchlist VM Options

Flipping over to the VM options, we have the ability to power off, suspend, reset, shutdown, or gracefully restart the VM. We also have some snapshot functionality to take a snapshot, or manage VM snapshots.

Additional Notes

In my environment I have two HPe DL360p Gen8 Servers and the sensor data is fully supported (I used the HPe custom ESXi install image which includes host drivers).

Aug 152018
 

As of yesterday, Samsung Pay is now available in Canada on your Samsung Gear S3 (and Gear Sport) smartwatch. You’ll need to update the Samsung Gear up to the new Galaxy Wearable app to enable the feature.

Samsung Pay for Samsung Gear S3 and Gear Sport

For some time users have been frustrated that while Samsung Pay has been available in Canada on your mobile devices, it hasn’t been available on their Samsung Gear smartwatches. This is no longer the case. While it has been released, some users are still reporting issues when trying to activate, such as the “Watch is not supported” error.

To get it working:

  1. Make sure your firmware is up to date on your watch using the Samsung Gear app (or the new Samsung Wearable app).
  2. Make sure your app is up to date. Samsung Gear should update to the new name of Samsung Wearable.
  3. Open “Samsung Wearable”, scroll down and look for “Samsung Pay” on the “Info” or “Settings” tab. Tap on this.
  4. The “Samsung Pay (Plug-in) should now download. Wait for it to launch.
  5. When opening Samsung Pay in the Samsung Wearable app, you may receive the error “Watch is not supported”.
  6. Go to “About Gear” under the “Settings” tab, and perform a full manual backup of your Samsung Gear. As soon as this completed, I received the notification on my watch on how to configure Samsung Pay.
  7. Press and hold the “Back” button on the watch, and follow the instructions. The Samsung Pay app should then install on your Samsung Gear watch.

I swear by Samsung pay (and use it with a number of my cards and loyalty programs). It’s awesome that it’s finally been released in Canada!

Aug 122018
 
VMware Horizon View Icon

On VMware Horizon view after updating the view agent on the VM, you may notice that USB redirection stops working with the error “USB Redirection is not available for this desktop”. This is due to an issue with the certificates on the VDI host (The VM running the VDI OS), after the VMware view agent upgrade is completed.

To resolve this you must use MMC, open the local computer certificate store, browse to “VMwareView\Certificates”, delete the agent certificates (for the local agent), and finally reboot for the agent to regenerate the certificates.

See below for instructions:

  1. While connected to the VM running the VDI OS, click Start, type “mmc.exe” (without quotations), and open the Microsoft Management Console.
    mmc.exe

    Open MMC by running mmc.exe

     

  2. Open the “Add/Remote Snap-in” wizard.

    Open the Add/Remove Snap-in Wizard

     

  3. We must now open the local certificate store on the local computer. Select “Certicates” on the Available Snap-ins, click “Add”, select “Computer Account”, then proceed to choose “Local Computer” and complete the wizard.

    Select the Computer account certificate store on the local computer

     

  4. Expand the “Certificates (Local Computer)” on the left underneath “Console Root”. Expand “VMwareView”, then expand and select “Certificates”. Select the certificate on the right that matches the local computer name of the VDI host, right click and select “Delete”. You may have to do this multiple times if multiple certificates exist for the local computer.

    Delete the VMwareView local agent certificate

     

  5. Restart the VDI host. And USB redirection should now be working!

    VMware View USB Redirection issue resolved

     

Cheers to VDI!

May 172018
 
Digitally Accurate Inc. Logo

Looking for Calgary IT Managed Services or Calgary IT Consulting Services? We can help!

My company Digitally Accurate Inc. (https://www.digitallyaccurate.com/) has been helping businesses for almost 12 years with their IT strategies.

Feel free to reach out via E-mail, telephone, or LinkedIn to see how I can help! You can also visit the “Hire Stephen Wagner” tab, and yes, you’ll get to meet me! 🙂

My company is also partnered with numerous companies and can design, configure, and sell solutions including the following:

  • VMware Solution Design and Licensing
  • Veeam (Veeam Backup and Replication, Veeam Availability Suite, Veeam Backup Essentials)
  • HPe Servers, Storage, Networking
  • Aruba Networking
  • Microsoft Licensing (Including Office 365)
  • Sophos (Including Sophos UTM, and Sophos XG appliances)
  • 10ZiG Zero Clients
  • Duo Security (MFA)
  • Symantec (including Symantec Protection Suite)
  • Redhat (including RHEL: Redhat Enterprise Linux Server and Workstation)
  • Eaton UPS (Eaton Uninterrupted Power Supply and other Eaton power equipment)

Again, feel free to reach out for more information!

May 112018
 
Veeam Logo

This morning I noticed that CentOS 7.5 (1804) was available for upgrade via yum. After upgrading my CentOS instance from 7.4 to 7.5 on Microsoft Azure, I noticed that when running a backup using the Veeam Linux Agent, the system would crash and become completely unresponsive. I would have to manually restart the OS.

Upon reboot, I analyzed the console messages log and ran the backup again to see what was happening.

Here’s a look at my /var/log/messages:

May 11 07:24:46 HOSTNAME kernel: Request for unknown module key 'Veeam Software AG: 9d063645550b483bec752cb3c0249d5ede714b3e' err -11
May 11 07:24:46 HOSTNAME kernel: veeamsnap: loading out-of-tree module taints kernel.
May 11 07:24:46 HOSTNAME kernel: WARNING: module 'veeamsnap' built without retpoline-enabled compiler, may affect Spectre v2 mitigation
May 11 07:24:46 HOSTNAME kernel: veeamsnap: module verification failed: signature and/or required key missing - tainting kernel
May 11 07:24:46 HOSTNAME kernel: veeamsnap: applying kernel_stack fix up
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init Loading
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init Version: 2.0.0.400
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init Author: Veeam Software AG
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init licence: GPL
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init description: Veeam Snapshot Kernel Module
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init zerosnapdata: 1
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init debuglogging: 0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init snapstore enabled
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init start. container_alloc_counter=0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init start. container_sl_alloc_counter=0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init start. mem_cnt=0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init start. vmem_cnt=0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:ctrl_pipe_init .
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init Module major=243
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:blk_direct_bioset_create Specific bio set created.
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:blk_redirect_bioset_create Specific bio set created.
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:blk_deferred_bioset_create Specific bio set created.
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:snapimage_init .
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:snapimage_init Snapimage block device was registered. major=252
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init end. container_alloc_counter=0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init end. container_sl_alloc_counter=0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init end. mem_cnt=1
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:veeamsnap_init end. vmem_cnt=0
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:ctrl_open file=0xffff95e4543b1800
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:ctrl_pipe_new .
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:ioctl_compatibility_flags Get compatibility flags
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:ioctl_tracking_collect Collecting tracking device:
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:tracking_collect Have not device under CBT.
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:tracking_add Adding. dev_id=8:1
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:tracker_Create dev_id 8:1
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:tracker_Create SectorStart    =0x800
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:tracker_Create SectorsCapacity=0xfa000
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:tracker_cbt_start .
May 11 07:24:46 HOSTNAME kernel:    veeamsnap:cbt_map_create CBT map create.
May 11 07:24:47 HOSTNAME kernel: general protection fault: 0000 [#1] SMP
May 11 07:24:47 HOSTNAME kernel: Modules linked in: veeamsnap(OE) nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner xt_conntrack nf_conntrack iptable_security ext4 mbcache jbd2 dm_mirror dm_region_hash dm_log dm_mod sb_edac iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr i2c_piix4 sg hv_utils i2c_core ptp pps_core hv_balloon ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi ata_piix hv_storvsc hv_netvsc libata scsi_transport_fc hid_hyperv hyperv_keyboard scsi_tgt hyperv_fb crct10dif_pclmul crct10dif_common crc32c_intel hv_vmbus floppy serio_raw
May 11 07:24:47 HOSTNAME kernel: CPU: 1 PID: 1712 Comm: Lpb Server thre Tainted: G           OE  ------------   3.10.0-862.2.3.el7.x86_64 #1
May 11 07:24:47 HOSTNAME kernel: Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007  06/02/2017
May 11 07:24:47 HOSTNAME kernel: task: ffff95e447378000 ti: ffff95e45cbe0000 task.ti: ffff95e45cbe0000
May 11 07:24:47 HOSTNAME kernel: RIP: 0010:[]  [] page_array_memset+0x4d/0xa0 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: RSP: 0018:ffff95e45cbe3d60  EFLAGS: 00010246
May 11 07:24:47 HOSTNAME kernel: RAX: 0000000000000000 RBX: ffff95e449615200 RCX: 0000000000000200
May 11 07:24:47 HOSTNAME kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff187288716000
May 11 07:24:47 HOSTNAME kernel: RBP: ffff95e45cbe3d60 R08: ffffffffbe274fef R09: ffff95e460affa60
May 11 07:24:47 HOSTNAME kernel: R10: ffff95e460affa60 R11: 0000000000000000 R12: 0000000000000001
May 11 07:24:47 HOSTNAME kernel: R13: 00000000000fa000 R14: 0000000000000000 R15: 0000000000800001
May 11 07:24:47 HOSTNAME kernel: FS:  00007f336f7fe700(0000) GS:ffff95e495640000(0000) knlGS:0000000000000000
May 11 07:24:47 HOSTNAME kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 11 07:24:47 HOSTNAME kernel: CR2: 0000000000738df0 CR3: 00000002d3afc000 CR4: 00000000001406e0
May 11 07:24:47 HOSTNAME kernel: Call Trace:
May 11 07:24:47 HOSTNAME kernel: [] cbt_map_allocate+0x6e/0x160 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: [] cbt_map_create+0x73/0x100 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: [] tracker_cbt_start+0x5a/0xc0 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: [] tracker_Create+0x16a/0x650 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: [] tracking_add+0x2e0/0x450 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: [] ioctl_tracking_add+0x6c/0x170 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: [] ctrl_unlocked_ioctl+0x4e/0x60 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: [] do_vfs_ioctl+0x350/0x560
May 11 07:24:47 HOSTNAME kernel: [] ? __sb_end_write+0x31/0x60
May 11 07:24:47 HOSTNAME kernel: [] ? vfs_write+0x182/0x1f0
May 11 07:24:47 HOSTNAME kernel: [] SyS_ioctl+0xa1/0xc0
May 11 07:24:47 HOSTNAME kernel: [] system_call_fastpath+0x1c/0x21
May 11 07:24:47 HOSTNAME kernel: Code: 01 49 89 f9 48 0f af c2 49 8b 79 10 ba 00 10 00 00 40 f6 c7 01 75 47 40 f6 c7 02 75 51 40 f6 c7 04 75 2b 89 d1 c1 e9 03 83 e2 07  48 ab 74 0e 41 89 c8 83 c1 01 39 d1 42 88 34 07 72 f2 49 83
May 11 07:24:47 HOSTNAME kernel: RIP  [] page_array_memset+0x4d/0xa0 [veeamsnap]
May 11 07:24:47 HOSTNAME kernel: RSP 
May 11 07:24:47 HOSTNAME kernel: ---[ end trace 96b51a664f4baef9 ]---

It appeared the veeamsnap module was causing a protection fault with the kernel, causing the system to crash.

In an effort to troubleshoot, I uninstalled and reinstalled Veeam and tried rebuilding the kernel module, however the issue still persisted. After some searching, I came across these two posts:

https://forums.veeam.com/veeam-agent-for-linux-f41/veeam-agent-for-linux-and-rhel-7-5-crash-t50170.html

https://www.veeam.com/kb2569

According to the KB, the veeamsnap module isn’t compatible with kernel version 3.10.0-862.

Checking my system after upgrading CentOS 7.5:

[root@HOSTNAME ~]# uname -a
Linux HOSTNAME.somedomain.com 3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9 18:05:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@HOSTNAME ~]# cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)

 

Essentially, as of right now the Veeam Agent for Linux is not yet supported on CentOS 7.5, RHEL 7.5, or Oracle RHCK 7.5. If you’re running any of these, hold off and do not install Veeam Agent for Linux. If you are scheduling an upgrade, do not perform upgrade unless you want to break your backup. It sounds like this should be fixed in a future update.

May 082018
 

Recently a customer of mine who is using an outdated version of Intuit QuickBooks on their Terminal Server (RDS Remote Desktop Services) started to experience an issue when users attempted to log on. QuickBooks would initialize, then prompt the TLS 1.2 warning message, and then completely crash. This would prevent the users from doing any tasks.

In an effort to troubleshoot this, I tried to use different accounts, checked the QBW.ini file for any flags that could dismiss this message, tried Intuit’s “TLS preparedness tool” (which still scares me because I have no idea what system changes it made on the server), etc… All of these attempts had no effect on the issue.

For a temporary workaround, you’ll need to load up QuickBooks on a workstation (not using terminal services). You’ll need to open the datafile, select “Do Not Show this again”, and then close. You’ll need to do this for each datafile. Please note that if you do not receive the prompts on other datafiles, you’ll need to open the datafile with a different Quickbooks username (QB account, not Windows account) in order to get it to prompt.

This issue should only occur on older versions of Quickbooks when using TS/RDS. To officially resolve this issue, I recommend upgrading to a more recent (and in support) version of Quickbooks.

May 062018
 
DUO

I’m a big fan of MFA, specifically Duo Security‘s product (I did a corporate blog post here). I’ve been using this product for some time and use it for an extra level of protection on my workstations, servers, and customer sites. I liked it so much so that my company (Digitally Accurate Inc.) became a partner and now resells the services.

Today I want to write about a couple issues I had when deploying the pam_duo module on CentOS Linux 7. The original duo guide can be found at https://duo.com/docs/duounix, however while it did work for the most part, I noticed there were some issues with the pam configuration files, especially if you are wanting to use Duo MFA with usernames and passwords, and not keys for authentication.

A symptom of the issue: I noticed that when following the instructions on the website for deployment, after entering the username, it would skip the password prompt, and go right for DUO authentication, completely bypassing the password all together. I’m assuming this is because the guide was written for key authentication, but I figured I’d do a quick crash-course post on the topic and create a simple guide. I also noticed that sometimes even if an incorrect password was typed in, it would allow authentication if DUO passed as successful.

Ultimately I decided to learn about PAM, understand what it was doing, and finally configure it properly. Using the guide below I can confirm the password and MFA authentication operate correctly.

To configure Duo MFA on CentOS 7 for use with usernames and passwords

First and foremost, you must log in to your Duo Account and go to applications, click “Protect an Application” and select “Unix Application”. Configure the application and document/log your ikey, secret key, and API hostname.

Now we want to create a yum repo where we can install, and keep the pam_duo module up to date. Create the file /etc/yum.repos.d/duosec.repo and then populate it with the following:

[duosecurity]
name=Duo Security Repository
baseurl=http://pkg.duosecurity.com/CentOS/$releasever/$basearch
enabled=1
gpgcheck=1

We’ll need to install the signging key that the repo uses, and then install the duo_unix package. By using yum, we’ll be able to keep this package regularly up to date when we update the server. Run the following commands:

rpm --import https://duo.com/RPM-GPG-KEY-DUO
yum install duo_unix

Configure the pam_duo module by editing the /etc/duo/pam_duo.conf file. You’ll need to populate the lines with your ikey, secret key, and API hostname that you documented above. We use “failmode=safe” so that in the event of an internet disconnection, we can still login to the server without duo. It’s safe to enable this fail-safe, as the purpose is to protect it against the internet. Please see below:

[duo]
; Duo integration key
ikey = XXXXXXXXXXXXXXXXXXXX
; Duo secret key
skey = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
; Duo API host
host = XXXXXXXXX.duosecurity.com
; Send command for Duo Push authentication
pushinfo = yes
; failmode safe if no internet it works (secure locks it up)
failmode = safe

Configure sshd to allow Challenge Response Authentication by editing /etc/ssh/sshd_config, then locate and change “ChallengeResponseAuthentication” to yes. Please note that the line should already be there, and you should simply have to move the comment symbol to comment the old line, and uncomment the below line as shown below:

ChallengeResponseAuthentication yes

And now it gets tricky… We need to edit the pam authentication files to incorporate the Duo MFA service in to it’s authentication process. I highly recommend that throughout this, you open (and leave open) an additional SSH session, so that if you make a change in error and lock yourself out, you can use the extra SSH session to revert any changes to the system to re-allow access. It’s always best to make a backup and copy of these files so you can easily revert if needed.

DISCLAIMER: I am not responsable if you lock yourself out of your system. Please make sure that you have an extra SSH session open so that you can revert changes. It is assumed you are aware of the seriousness of the changes you are making and that you are taking all precautions (including a backup) to protect yourself from any errors.

Essentially two files are used for authentication that we need to modify. One file is for SSH logins, and the other is for console logins. In my case, I wanted to protect both methods. You can do either, or both. If you are doing both, it may be a good idea to test with SSH, before making modifications to your console login, to make sure your settings are correct. Please see below for the modifications to enable pam_duo:

/etc/pam.d/password-auth (this file is used for SSH authentication)

#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth        required      pam_env.so
auth        required      pam_faildelay.so delay=2000000
#auth        sufficient    pam_unix.so nullok try_first_pass
auth        requisite     pam_unix.so nullok try_first_pass
auth  sufficient pam_duo.so
auth        requisite     pam_succeed_if.so uid >= 1000 quiet_success
auth        required      pam_deny.so

account     required      pam_unix.so
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 1000 quiet
account     required      pam_permit.so

password    requisite     pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type=
password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok


password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
-session     optional      pam_systemd.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so

/etc/pam.d/system-auth (this file is used for console authentication)

auth        required      pam_env.so
auth        sufficient    pam_fprintd.so
#auth        sufficient    pam_unix.so nullok try_first_pass
# Next two lines are for DUO mod
auth        requisite     pam_unix.so nullok try_first_pass
auth        sufficient    pam_duo.so
auth        requisite     pam_succeed_if.so uid >= 1000 quiet_success
auth        required      pam_deny.so

account     required      pam_unix.so
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 1000 quiet
account     required      pam_permit.so

password    requisite     pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type= ucredit=-1 lcredit=-1 dcredit=-1 ocredit=-1
password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok remember=5
password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
-session     optional      pam_systemd.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so

Now, we must restart sshd for the changes to take affect. Please make sure you have your extra SSH session open in the event you need to rollback your /etc/pam.d/ files. Restart the sshd service using the following command:

service sshd restart

Attempt to open a new SSH session to your server. It should now ask for a username, password, and then prompt for Duo authentication. And you’re done!

More information on Duo Multi Factor Authentication (MFA) can be found here.