May 052019
 
Ubuntu Orange Logo

After upgrading a computer from Ubuntu 16.04 LTS to Ubuntu 18.04 LTS, during boot the screen goes blank (turns black), all HD disk activity halts, and the system becomes frozen. This event can also occur on a fresh installation or when updates are installed.

This is due to a video mode issue that causes the system to halt or freeze. It’s much like the issue I described here on a Fedora Linux system.

Temporary Fix

To get the system to boot:

  1. After turning on your PC, hold the right SHIFT key to get to the GRUB bootloader.
  2. Once GRUB is open, press the “e” key to edit the first highlighted entry “Ubuntu”.
  3. Move your cursor down to the line that starts with “linux”, and use the right arrow key to find the section with the words “ro quiet splash”.
  4. Add “nomodeset” after these words.
    nomodeset
  5. Feel free to remove “quiet” and “splash” for more verbosity to troubleshoot the boot process.
  6. Press “CTRL + X” or “F10” to boot.
  7. The system should now boot.

Permanent Fix

To permanently resolve the issue:

  1. Once the system has booted using the temporary fix, log in.
  2. Open a terminal window (Applications -> Terminal, or press the “Start” button and type terminal).
  3. Either “su” in to root, or use “sudo” to open your favorite text editor and edit the file “/etc/default/grub” (I use nano which can be install by running “dnf install nano”):
    nano /etc/default/grub
  4. Locate the line with the variable “GRUB_CMDLINE_LINUX_DEFAULT”, and add “nomodeset” to the variables. Feel free to remove “splash” and “quiet” if you’d like text boot. Here’s an example of my line after editing (yours will look different):
    GRUB_CMDLINE_LINUX_DEFAULT="quiet splash nomodeset"
  5. Save the file and exit the text editor (CTRL+X to quit, the press “y” and enter to save).
  6. At the bash prompt, execute the following command to regenerate the grub.conf file on the /boot partition from your new default file:
    update-grub
  7. Restart your system, it should now boot!

Please Note: Always make sure you have a full system backup before modifying any system files!

May 042019
 
Ubuntu Orange Logo

You’re trying to install Ubuntu on your computer, but it freezes due to lack of resources, specifically memory. This can happen when you’re trying to re-purpose old laptops, netbooks, etc.

This recently happened to met as I tried to install Ubuntu on an old HP Netbook. Originally I used Fedora, but had to switch to Ubuntu due to library issues (I wanted to use the VMware Horizon Client on it).

Unfortunately, when I’d kick off the USB installer, the OS would completely freeze (mouse either unresponsive, or extremely glitchy).

The Fix – External SWAP File

In the ~5 minutes where the system is operable, I used the key sequence “CTRL + ALT + F2” to get to a text tty console session. From here I noticed the system eventually uses all the RAM and maxes out the memory. When this occurs, this is when the system becomes unresponsive.

Since this is a Live CD installer, there is no swap file for the system to use once the RAM has filled up.

To fix this and workaround the problem, I grabbed a second blank USB stick and used it as an external swap file. Using this allowed me to run the installer, complete the installer, and successfully install Ubuntu.

Please make sure you are choosing the right device names in the instructions below. Choosing the wrong device name can cause your to write to the wrong USB stick, or worse the hard drive of your system.

Instructions:

  1. Attached USB Installer, boot system.
  2. Once system has booted, press “CTRL + ALT + F2” to open a tty console session.
  3. Login using user: “Ubuntu” with a blank password.
  4. Type “sudo su” to get a root shell.
  5. Type in “tail -f /var/log/kern.log” and connect your spare blank USB stick that you want to use for SWAP space. Note the device name, in my case it was “/dev/sdd”.
  6. Press “CTRL + C” to stop tailing the log file, then run “fdisk /dev/sdd” and replace “/dev/sdd” with whatever your device was. PLEASE MAKE SURE YOU ARE CHOOSING THE RIGHT USB DEVICE NAME.
  7. Use “n” to create a new partition, follow the prompts, when it asks for size I randomly chose “+2G” for a 2GB swap file. Use “w” to write the partition table and then quit the fdisk application.
  8. Run “mkswap /dev/sdd1” and replace “sdd1” with the device and partition number of your USB Swap stick. This will format the partition and mark it as a SWAP filesystem.
  9. Run “swapon /dev/sdd1” and replace “sdd1” with your swap partition you created. This will activate the external swap file on the USB stick.
  10. Press “CTRL + ALT + F1” to return to the Ubuntu installation guide. Continue the install as normal.

This should also work for other Linux distributions, as I have also used this in the past with Fedora (on a Single Board Computer with almost no RAM).

During the install process where the Ubuntu installer formats your hard drive, the install will actually mount the hard drive swap file as well (it’ll use both). Once the installer is complete, shut down the system and remove the USB SWAP stick.

May 032019
 
Ubuntu Enterprise Logo

So here we’re going to talk about Installing the VMware Horizon Agent for Linux on an Ubuntu 18.04 Guest VM for use with VDI. Ultimately you’ll be setting up Horizon 7 for Linux Desktops.

This will allow you to add Linux VMs to your VDI environment.

Ubuntu 18.04 LTS running on VMware Hoirzon Client using Horizon for Linux
Horizon for Linux on Ubuntu 18.04 LTS

VMware has the documentation here, but I’ve condensed this down for your to get up and running quickly, as well as deal with a few bugs I noticed.

Requirements

You’ll need the following to get started

  • VMware Horizon View 7 (I’m, using 7.8)
  • Horizon Enterprise or Horizon for Linux Valid Licensing
  • Horizon VDI environment that’s functioning and working
  • Ubuntu 18.04 LTS Installer ISO (download here)
  • Horizon Agent for Linux (download here)
  • Functioning internal DNS

Once you have the above, we can get going!

Instructions

  1. Create a static DHCP reservation, or note the IP you will set statically to the Ubuntu VM
  2. Create a host entry on your internal DNS Server so that the IP of your Ubuntu VM will have a functioning internal FQDN
  3. Create a VM and Install Ubuntu 18.04 TLS using the ISO you downloaded above
  4. If you’re using a static IP, set this now. If you’re using a DHCP reservation, make sure it’s working
  5. Install any Root CA’s or modifications you need for network access (usually not needed unless you’re on an enterprise network)
  6. Open a terminal, and type “sudo su” to get a root console
  7. Run the following command
    apt install open-vm-tools-desktop openssh-server python python-dbus python-gobject lightdm
    You can skip the “openssh-server” package if you don’t want to enable SSH. A display manager configuration prompt will present itself, choose “gdm”
  8. Now we need to add the internal FQDN to the hosts file. Run “nano /etc/hosts” to open the hosts file. Create a new line at the top and enter
    127.0.0.1 compname.domain.com compname
    Modify “compname.domain.com” and “compname” to reflect your FQDN and computer name.
  9. Restart the Guest VM
  10. Open terminal, “sudo su” to get a root console
  11. Extract the Horizon Agent tarball with
    tar zxvf VMware-horizonagent-linux-x86_64-7.8.0-12610615.tar.gz
    Please note that if your version is different, your file name may be different. Please adjust accordingly.
  12. Change directory in to the VMware Horizon Agent that we just extracted.
  13. Run the installer for the horizon agent with
    ./install_viewagent.sh
  14. Follow the prompts, restart the host
  15. Log on to your View Connection Server
  16. Create a manual pool, and configure it accordingly
  17. Add the Ubuntu Linux VM to the pool
  18. Entitle the users to the pool, and assign the users to the host under inventory

Final Notes

In the VMware documentation, it states to select “lightdm” on the Display MAnager configuration window that presents itself in step 7. However if you choose this, the VMware Horizon Agent for Linux will not install. Choosing “gdm” allows it to install and function.

I have noticed audio issues when using the Spotify snapd. I believe this is caused by timer-based audio scheduling in PulseAudio. I have tried using the “tsched=0” flag in the PulseAudio config, however this has no effect and I haven’t been able to resolve this yet. Audio in Chrome and other audio players works fine. A workaround is to install “pavucontrol” and have it open while using Spotify and the audio issues will temporarily be resolved. I also tried using the VMware Tools (deprecated) instead of OpenVM Tools to see if this helped with the audio issues, but it did not.

If you have 3D Acceleration with a GRID card, the Linux VDI VM will be able to utilize 3D accelerated vSGA as long as you have it configured on the ESXi host.

Aug 262018
 
Fedora Logo

One of the coolest things I love about running VMware Horizon View and VDI is that you can repurpose old computers, laptops, or even netbooks in to perfect VDI clients running Linux! This is extremely easy to do and gives life to old hardware you may have lying around (and we all know there’s nothing wrong with that).

I generally use Fedora and the VMware Horizon View Linux client to accomplish this. See below to see how I do it!

 

Quick Guide

  1. Download the Fedora Workstation install or netboot ISO from here.
  2. Burn it to a DVD/CD if you have DVD/CD drive, or you can write it to a USB stick using this method here.
  3. Install Fedora on to your laptop/notebook/netbook using the workstation install.
  4. Update your Fedora Linux install using the following command
    dnf -y upgrade
  5. Install the prerequisites for the VMware Horizon View Linux client using these commands
    dnf -y install https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
    dnf -y install gstreamer-plugins-ugly gstreamer-plugins-bad gstreamer-ffmpeg xine-lib-extras-freeworld xine-lib-extras-freeworld libssl* libcrypto* openssl-devel libpng12 systemd-devel libffi-devel
    
  6. To fix an issue with package versions and dependancies, run the following commands
    ln -s /usr/lib64/libudev.so.1 /usr/lib64/libudev.so.0
    ln -s /usr/lib64/libffi.so.6 /usr/lib64/libffi.so.5
  7. Download the VMware Horizon View Linux client from here
  8. Make the VMware bundle executable and then run the installer using these commands (your file name may be different depending on build version number)
    chmod 777 VMware-Horizon-Client-4.8.0-8518891.x64.bundle
    sudo ./VMware-Horizon-Client-4.8.0-8518891.x64.bundle
  9. Complete the installation wizard
  10. You’re done!

To run the client, you can find it in the GUI applications list as “VMware Horizon Client”, or you can launch it by running “vmware-view”.

VMware Horizon View on Linux in action

Here is a VMware Horizon View Linux client running on HP Mini 220 Netbook

Additional Notes:

-If you’re comfortable, instead of the workstation install, you can install the Fedora LXQt Desktop spin, which is a lightweight desktop environment perfect for low performance hardware or netbooks. More information and the download link for Fedora LXQt Desktop Spin can be found here: https://spins.fedoraproject.org/en/lxqt/

-If you installed Fedora Workstation and would like to install the LXQt window manager afterwards, you can do so by running the following command (after installing, at login prompt, click on the gear to change window managers):

dnf install @lxqt-desktop-environment

-Some of the prerequisites above in the guide may not be required, however I have installed them anyways for compatibility.

Aug 252018
 
Fedora Logo

After doing a fresh install or upgrade of Fedora Core Linux (FC28 in my case), you may notice that when the system boots it gets stuck on a black screen with a white cursor. The cursor will not move and there will be no drive activity.

This issue occurs with GNOME on my old HP Mini 210 Netbook every time I do a fresh install of Fedora on it (or upgrade it).

Follow the process below to temporarily boot and then permanently fix it.

Temporary fix

To get the system to boot:

  1. Power on the computer, and carefully wait for the GRUB bootloader to appear (the boot selection screen).
  2. When the GRUB bootloader appears, press the “e” key to edit the highlighted (default) boot entry.
  3. Scroll down until you get to the line starting with “linux16”, then use your right arrow key and scroll right until you get to the end of the kernel options (while scrolling right, you may scroll multiple lines down which is fine and expected). The line should finally end with “rhgb” and “quiet”.
  4. Remove “rhgb” and “quiet”, and then add “nomodeset=0”
  5. Press “CTRL+x” to boot the system.
  6. The system should now boot.

FYI: “rhgb” is the kernel switch/option for redhat graphical boot, and “quiet” makes the system messages more quiet (who would have guessed).

Permanent Fix

To permanently resolve the issue:

  1. Once the system has booted, log in.
  2. Open a terminal window (Applications -> Terminal, or press the “Start” button and type terminal).
  3. Use your favorite text editor and edit the file “/etc/default/grub” (I use nano which can be install by running “dnf install nano”):
    nano /etc/default/grub
  4. Locate the line with the variable “GRUB_CMDLINE_LINUX”, and add “nomodeset=0” to the variables. Feel free to remove “rhgb” and “quiet” if you’d like text boot. Here’s an example of my line after editing (yours will look different):
    GRUB_CMDLINE_LINUX="resume=/dev/mapper/fedora_da--netbook01-swap rd.lvm.lv=fedora_da-netbook01/root rd.lvm.lv=fedora_da-netbook01/swap nomodeset=0"
  5. Save the file and exit the text editor (CTRL+x to quit, the press “y” and enter to save)
  6. At the bash prompt, execute the following command to regenerate the grub.conf file on the /boot partition from your new default file:
    grub2-mkconfig -o /boot/grub2/grub.cfg
  7. Restart your system, it should now boot!

 

Please Note: Always make sure you have a full system backup before modifying any system files!

Aug 252018
 
Fedora Logo

A fun fact that a lot of users still aren’t aware of, is that you can create Fedora and CentOS bootable media (a bootable USB stick) using the DVD/CD ISO image, and using the linux dd command.

So if you have an existing and running Linux install, you can use this method to quickly write an ISO file to a USB stick!

 

Here’s How!

  1. Get your USB stick handy, make sure it’s big enough to store the ISO file you want to download.
  2. Download your preferred ISO DVD or CD Image for Installation from CentOS or Fedora.
  3. Connect your USB stick, open a terminal session and run the following command to identify the device name of the USB stick (mine was sdb for /dev/sdb):
    [root@StephenW-X1 ~]# dmesg | grep removable
    [  171.890670] sd 1:0:0:0: [sdb] Attached SCSI removable disk
  4. Issue the following command to write the ISO image to the USB stick. Change the input filename, and output device name to reflect your own.
    [root@StephenW-X1 Downloads]# dd if=Fedora-Workstation-netinst-x86_64-28-1.1.iso of=/dev/sdb
    1193984+0 records in
    1193984+0 records out
    611319808 bytes (611 MB, 583 MiB) copied, 13.6777 s, 44.7 MB/s
  5. Your done!

Please Note:

  • Choosing the wrong /dev/sd[x] device can case you to write the ISO file to your hard drive, or another hard drive in your system. Make sure you select the right device name. If you’re unsure, don’t run the command.
  • You can also use the “Fedora Media Writer for Windows” available here: https://getfedora.org/fmw/FedoraMediaWriter-win32-4.1.1.exe to write an ISO image to USB if you’re running Windows.
Aug 182018
 
CentOS Logo

Let’s say that you’re hosting someone’s equipment and they start to abuse their connection speed. Let’s say that you’re limited in your bandwidth, and you want to control your own bandwidth to make sure you don’t max out your own internet connection. You can take care of both of these problems by building your own traffic shaping network control device using CentOS and using the “tc” linux command.

In this post I’m going to explain what traffic shaping is, why you’d want to use traffic shaping, and how to build a very basic traffic shaping device to control bandwidth on your network.

What is traffic shaping

Traffic shaping is when one attempts to control a connection in their network to prioritize, control, or shape traffic. This can be used to control either bandwidth or packets. In this example we are using it to control bandwidth such as upload and download speeds.

Why traffic shaping

For service providers, when hosting customer’s equipment, the customer may abuse their connection or even max it out legitimately. This can put a halt on the internet connection if you share it with them, or cause bigger issues if it’s shared with other customers. In this example, you would want to implement traffic shaping to allot only a certain amount of bandwidth so they wouldn’t bring the internet connection or network to a halt.

For normal people (or a single business), as fast as the internet is today, it’s still very easy to max your connection out. When this happens you can experience packet loss, slow speeds, and interruption of services. If you host your own servers this can cause even a bigger issue with interruption of those services as well. You may want to limit your own bandwidth to make sure that you don’t bring your internet to a halt, and save some for other devices and/or users.

Another reason is just to implement basic QoS (Quality of Service) across your network, to keep usage and services in harmony and eliminate any from hogging the network connections up.

How to build your own basic traffic shaping device with CentOS and tc

In this post we will build a very simple traffic shaping device that limits and throttles an internet connection to a defined upload and download speed that we set.

You can do this with a computer with multiple NICs (preferably one NIC for management, one NIC for internet, and one NIC for network and/or the hosts to be throttled). If you want to get creative, there are also a number of physical network/firewall appliances that are x86 based, that you can install Linux on. These are very handy as they come with many NICs.

When I set this up, I used an old decommissioned Sophos UTM 220 that I’ve had sitting around doing nothing for a couple years (pic below). The UTM 220 provides 8 NICs, and is very easy to install Linux on to.

Sophos UTM 220 Running CentOS Linux

Sophos UTM 220 Running CentOS Linux

Please Note: The Sophos UTM 220 is just a fancy computer in a 1U rack mounted case with 8 NICs. All I did was install CentOS on it like a normal computer.

Essentially, all we’ll be doing is installing CentOS Linux, installing “tc”, configuring the network adapters, and then configuring a startup script. In my example my ISP provides me 174Mbps download, and 15Mbps upload. My target is to throttle the connection to 70Mbps download, and 8Mbps upload. I will allow the connection to burst to 80Mbps down, and 10Mbps up.

To get started:

  1. Install CentOS on the computer or device. The specifics of this are beyond the scope of this document, however you’ll want to perform a minimal install. This device is strictly acting as a network device, so no packages are required other than the minimal install option.
  2. During the CentOS install, only configure your main management NIC. This is the NIC you will use to SSH to, control the device, and update the device. No other traffic will pass through this NIC.
  3. After the install is complete, run the following command to enable ssh on boot:
    chkconfig sshd on
  4. Install “tc” by running the command:
    yum install tc
  5. Next, we’ll need to locate the NIC startup scripts for the 2 adapters that will perform the traffic shaping. These adapters are the internet NIC, and the NIC for the throttled network/hosts. Below is an example of one of the network startup scripts. You’re NIC device names will probably be different.
    /etc/sysconfig/network-scripts/ifcfg-enp2s0
  6. Now you’ll need to open the file using your favorite text editor and locate and set ONBOOT to no as shown below. You can ignore all the other variables. You’ll need to repeat this for the 2nd NIC as well.
    TYPE=Ethernet
    PROXY_METHOD=none
    BROWSER_ONLY=no
    BOOTPROTO=dhcp
    DEFROUTE=yes
    IPV4_FAILURE_FATAL=no
    IPV6INIT=yes
    IPV6_AUTOCONF=yes
    IPV6_DEFROUTE=yes
    IPV6_FAILURE_FATAL=no
    IPV6_ADDR_GEN_MODE=stable-privacy
    NAME=enp2s0
    UUID=xxxxxxxx-xxxx-xxx-xxxx-xxxxxxxxxxxx
    DEVICE=enp2s0
    ONBOOT=no
  7. Now we can configure the linux startup script to configure a network bridge between the two NICs above, and then configure the traffic shaping rules with tc. Locate and open the following file for editing:
    /etc/rc.d/rc.local
  8. Append the following text to the rc.local file:
    # Lets make that bridge
    brctl addbr bridge0
    
    # Lets add those NICs to the bridge
    brctl addif bridge0 enp5s0
    brctl addif bridge0 enp2s0
    
    # Confirm no IP set to NICs that are shaping
    ifconfig enp5s0 0.0.0.0
    ifconfig enp2s0 0.0.0.0
    
    # Bring the bridge online
    ifconfig bridge0 up
    
    # Clear out any existing tc policies
    tc qdisc del dev enp2s0 root
    tc qdisc del dev enp5s0 root
    
    # Configure new traffic shaping policies on the NICs
    # Set the upload to 8Mbps and burstable to 10mbps
    tc qdisc add dev enp2s0 root tbf rate 8mbit burst 10mbit latency 50ms
    # Set the download to 70Mbps and burstable to 80Mbps
    tc qdisc add dev enp5s0 root tbf rate 70mbit burst 80mbit latency 50ms
    
  9. Restart the linux box:
    shutdown -r now
  10. You now have a traffic shaping network device!

Final Thoughts

Please note that normally you would not place the script in the rc.local file, however we wanted something quick and simple. The script may not survive in the rc.local file when updates/upgrades are applied against on the Linux install, so keep this in mind. You’ll also need to test to make sure that you are throttling in the correct direction with the 2 NICs. Make sure you test this setup and allow time to confirm it’s working before putting it in a production network.

May 062018
 
DUO

I’m a big fan of MFA, specifically Duo Security‘s product (I did a corporate blog post here). I’ve been using this product for some time and use it for an extra level of protection on my workstations, servers, and customer sites. I liked it so much so that my company (Digitally Accurate Inc.) became a partner and now resells the services.

Today I want to write about a couple issues I had when deploying the pam_duo module on CentOS Linux 7. The original duo guide can be found at https://duo.com/docs/duounix, however while it did work for the most part, I noticed there were some issues with the pam configuration files, especially if you are wanting to use Duo MFA with usernames and passwords, and not keys for authentication.

A symptom of the issue: I noticed that when following the instructions on the website for deployment, after entering the username, it would skip the password prompt, and go right for DUO authentication, completely bypassing the password all together. I’m assuming this is because the guide was written for key authentication, but I figured I’d do a quick crash-course post on the topic and create a simple guide. I also noticed that sometimes even if an incorrect password was typed in, it would allow authentication if DUO passed as successful.

Ultimately I decided to learn about PAM, understand what it was doing, and finally configure it properly. Using the guide below I can confirm the password and MFA authentication operate correctly.

To configure Duo MFA on CentOS 7 for use with usernames and passwords

First and foremost, you must log in to your Duo Account and go to applications, click “Protect an Application” and select “Unix Application”. Configure the application and document/log your ikey, secret key, and API hostname.

Now we want to create a yum repo where we can install, and keep the pam_duo module up to date. Create the file /etc/yum.repos.d/duosec.repo and then populate it with the following:

[duosecurity]
name=Duo Security Repository
baseurl=http://pkg.duosecurity.com/CentOS/$releasever/$basearch
enabled=1
gpgcheck=1

We’ll need to install the signging key that the repo uses, and then install the duo_unix package. By using yum, we’ll be able to keep this package regularly up to date when we update the server. Run the following commands:

rpm --import https://duo.com/RPM-GPG-KEY-DUO
yum install duo_unix

Configure the pam_duo module by editing the /etc/duo/pam_duo.conf file. You’ll need to populate the lines with your ikey, secret key, and API hostname that you documented above. We use “failmode=safe” so that in the event of an internet disconnection, we can still login to the server without duo. It’s safe to enable this fail-safe, as the purpose is to protect it against the internet. Please see below:

[duo]
; Duo integration key
ikey = XXXXXXXXXXXXXXXXXXXX
; Duo secret key
skey = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
; Duo API host
host = XXXXXXXXX.duosecurity.com
; Send command for Duo Push authentication
pushinfo = yes
; failmode safe if no internet it works (secure locks it up)
failmode = safe

Configure sshd to allow Challenge Response Authentication by editing /etc/ssh/sshd_config, then locate and change “ChallengeResponseAuthentication” to yes. Please note that the line should already be there, and you should simply have to move the comment symbol to comment the old line, and uncomment the below line as shown below:

ChallengeResponseAuthentication yes

And now it gets tricky… We need to edit the pam authentication files to incorporate the Duo MFA service in to it’s authentication process. I highly recommend that throughout this, you open (and leave open) an additional SSH session, so that if you make a change in error and lock yourself out, you can use the extra SSH session to revert any changes to the system to re-allow access. It’s always best to make a backup and copy of these files so you can easily revert if needed.

DISCLAIMER: I am not responsable if you lock yourself out of your system. Please make sure that you have an extra SSH session open so that you can revert changes. It is assumed you are aware of the seriousness of the changes you are making and that you are taking all precautions (including a backup) to protect yourself from any errors.

Essentially two files are used for authentication that we need to modify. One file is for SSH logins, and the other is for console logins. In my case, I wanted to protect both methods. You can do either, or both. If you are doing both, it may be a good idea to test with SSH, before making modifications to your console login, to make sure your settings are correct. Please see below for the modifications to enable pam_duo:

/etc/pam.d/password-auth (this file is used for SSH authentication)

#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth        required      pam_env.so
auth        required      pam_faildelay.so delay=2000000
#auth        sufficient    pam_unix.so nullok try_first_pass
auth        requisite     pam_unix.so nullok try_first_pass
auth  sufficient pam_duo.so
auth        requisite     pam_succeed_if.so uid >= 1000 quiet_success
auth        required      pam_deny.so

account     required      pam_unix.so
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 1000 quiet
account     required      pam_permit.so

password    requisite     pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type=
password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok


password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
-session     optional      pam_systemd.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so

/etc/pam.d/system-auth (this file is used for console authentication)

auth        required      pam_env.so
auth        sufficient    pam_fprintd.so
#auth        sufficient    pam_unix.so nullok try_first_pass
# Next two lines are for DUO mod
auth        requisite     pam_unix.so nullok try_first_pass
auth        sufficient    pam_duo.so
auth        requisite     pam_succeed_if.so uid >= 1000 quiet_success
auth        required      pam_deny.so

account     required      pam_unix.so
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 1000 quiet
account     required      pam_permit.so

password    requisite     pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type= ucredit=-1 lcredit=-1 dcredit=-1 ocredit=-1
password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok remember=5
password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
-session     optional      pam_systemd.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so

Now, we must restart sshd for the changes to take affect. Please make sure you have your extra SSH session open in the event you need to rollback your /etc/pam.d/ files. Restart the sshd service using the following command:

service sshd restart

Attempt to open a new SSH session to your server. It should now ask for a username, password, and then prompt for Duo authentication. And you’re done!

More information on Duo Multi Factor Authentication (MFA) can be found here.

May 012018
 
Fedora Logo

When attempting to upgrade from Fedora Core 27 to Fedora Core 28, it may fail on the nss-pem package.

 

I spent some time trying to find the solution for this, and came across numerous posts on the “Red Hat Bugzilla”, particularly this post.

See below example:

[root@SYSTEMZ01 ~]# dnf system-upgrade download --releasever=28
Before you continue ensure that your system is fully upgraded by running "dnf --refresh upgrade". Do you want to continue [y/N]: y
Fedora 28 - x86_64 - Updates                                                                                                                                $
Fedora 28 - x86_64                                                                                                                                          $
google-chrome                                                                                                                                               $
RPM Fusion for Fedora 28 - Free - Updates                                                                                                                   $
RPM Fusion for Fedora 28 - Free                                                                                                                             $
RPM Fusion for Fedora 28 - Nonfree - Updates                                                                                                                $
RPM Fusion for Fedora 28 - Nonfree                                                                                                                          $
skype (stable)                                                                                                                                              $
Last metadata expiration check: 0:00:00 ago on Tue 01 May 2018 04:28:04 PM MDT.
Error:
 Problem: nss-pem-1.0.3-6.fc27.i686 has inferior architecture
  - nss-pem-1.0.3-6.fc27.x86_64 does not belong to a distupgrade repository
  - problem with installed package nss-pem-1.0.3-6.fc27.i686

To resolve this, manually install the nss-pem packages from FC28 prior to the upgrade using the following command.

dnf install nss-pem-1.0.3-9.fc28 --releasever=28

After doing so, re-attempt to upgrade and the upgrade should now proceed.

Jan 182018
 

The Problem

I run a Sophos UTM firewall appliance in my VMware vSphere environment and noticed the other day that I was getting warnings on the space used on the ESXi host for the thin-provisioned vmdk file for the guest VM. I thought “Hey, this is weird”, so I enabled SSH and logged in to check my volumes. Everything looked fine and my disk usage was great! So what gives?

After spending some more time troubleshooting and not finding much, I thought to myself “What if it’s not unmapping unused blocks from the vmdk to the host ESXi machine?”. What is unampping you ask? When files get deleted in a guest VM, the free blocks aren’t automatically “unmapped” and released back to the host hypervisor in some cases.

Two things need to happen:

  1. The guest VM has to release these blocks (notify the hypervisor that it’s not using them, making the vmdk smaller)
  2. The host has to reclaim these and issue the unmap command to the storage (freeing up the space on the SAN/storage itself)

On a side note: In ESXi 6.5 and when using VMFS version 6 (VMFS6), the datastores can be configured for automatic unmapping. You can still kick it off manually, but many administrators would prefer it to happen automatically in the background with low priority (low I/O).

Most of my guest VMs automatically do the first step (releasing the blocks back to the host). On Windows this occurs with the defrag utility which issues trim commands and “trims” the volumes. On linux this occurs with the fstrim command. All my guest VMs do this automatically with the exception being the Sophos UTM appliance.

The fix

First, a warning: Enable SSH on the Sophos UTM at your own risk. You need to know what you are doing, this also may pose a security risk and should be disabled once your are finished. You’ll need to “su” to root once you log in with the “loginuser” account.

This fix not only applies to the Sophos UTM, but most other linux based guest virtual machines.

Now to fix the issue, I used the “df” command which provides a list of the filesystems, their mount points, and storage free for those fileystems. I’ve included an example below (this wasn’t the full list):

hostname:/root # df
Filesystem                       1K-blocks     Used Available Use% Mounted on
/dev/sda6                          5412452  2832960   2281512  56% /
udev                               3059712       72   3059640   1% /dev
tmpfs                              3059712      100   3059612   1% /dev/shm
/dev/sda1                           338875    15755    301104   5% /boot
/dev/sda5                         98447760 13659464  79513880  15% /var/storage
/dev/sda7                        129002700  4624468 117474220   4% /var/log
/dev/sda8                          5284460   274992   4717988   6% /tmp
/dev                               3059712       72   3059640   1% /var/storage/chroot-clientlessvpn/dev


You’ll need to run the fstrim command on every mountpoint for file systems “/dev/sdaX” (X means you’ll be doing this for multiple mountpoints). In the example above, you’ll need to run it on “/”, “/boot”, “/var/storage”, “/var/log”, “/tmp”, and any other mountpoints that use “/dev/sdaX” filesytems.

Two examples:

fstrim / -v

fstrim / -v

 

 

fstrim /var/storage -v

fstrim /var/storage -v

 

 

Again, you’ll repeat this for all mount points for your /dev/sdaX storage (X is replaced with the volume number). The command above only works with mountpoints, and not the actual device mappings.

Time to release the unused blocks to the SAN:

The above completes the first step of releasing the storage back to the host. Now you can either let the automatic unmap occur slowly overtime if you’re using VMFS6, or you can manually kick it off. I decided to manually kick it off using the steps I have listed at: https://www.stephenwagner.com/2017/02/07/vmfs-unmap-command-on-vsphere-6-5-with-vmfs-6-runs-repeatedly/

You’ll need to use esxcli to do this. I simply enabled SSH on my ESXi hosts temporarily.

Please note: Using the unmap command on ESXi hosts is very storage I/O intensive. Do this during maintenance window, or at a time of low I/O as this will perform MAJOR I/O on your hosts…

I issue the command (replace “DATASTORENAME” with the name of your datastore):

esxcli storage vmfs unmap --volume-label=DATASTORENAME --reclaim-unit=8

This could run for hours, possibly days depending on your “reclaim-unit” size (this is the block size of the unit you’re trying to reclaim from the VMFS file-system). In this example I choose 8, but most people do something larger like 100, or 200 to reduce the load and time for the command to complete (lower values look for smaller chunks of free space, so the command takes longer to execute).

I let this run for 2 hours on a 10TB datastore, however it may take way longer (possibly 6+ hours, to days).

Finally, not only are we are left with a smaller vmdk file, but we’ve released the space back to the SAN as well!