In this tutorial, I will be showing you how to get Lio-Target (an iSCSI target that is compatible with persistent reservations required by both VMware and MS Clustering) running on CentOS 6.
While this tutorial is targetted for CentOS 6 users, I see no reason why this should work on any other newer distributions.
Please note that while Lio-Target 4.x (and required tcm_loop and iSCSI) is available on newer/non-stable development kernels, Lio 3.X is stable, and currently builds nicely on CentOS 6. I will be doing up a tutorial for Lio 4.X once I myself start using it.
One more note, In the past I have thrown up a few tutorials on how to get Lio-Target running on various Linux distributions. These tutorials have worked for some, and not for others. I myself have had a few difficulties replicating the success I did originally. I myself am a technical guy, I do not understand some developer terms, and am not an expert in understanding some development cycles. This is one of the reasons why I had so many difficulties earlier. Since the earlier tutorials, I have caught up to speed and am familiar with what is required to get Lio-Target running.
Now on to the tutorial:
It is a good idea to start with a fresh install of CentOS 6. Make sure you do not have any of the iSCSI target packages installed that ship with CentOS. In my case I had to remove a package called something like “iSCSI-Target-utils” (This shipped with the CentOS 6 install).
1. Let’s download the software. We need to download both the 3.5 version of Lio-Target, along with Lio-utils which was built for 3.x of Lio-Target. (I chose the RisingTide Systems GIT repo since lio related projects have been missing from kernel.org’s GIT repo due to the issues kernel.org has been having recently).
Issue the following commands:
git clone git://risingtidesystems.com/lio-core-backports.git lio-core-backports.git
git clone git://risingtidesystems.com/lio-utils.git lio-utils.git
cd lio-utils.git/
git checkout --track -b lio-3.5 origin/lio-3.5
cd ..
(You have now downloaded both Lio-Target 3.5 backport, and lio-utils for lio-target 3.x)
2. Build kernel modules for your existing running CentOS kernel. Make sure you change in to the lio-core-backports directory first.
Change in to the lio-core-backports directory then issue the following commands:
make
make install
(You have now built, and installed the kernel modules for Lio-Target)
3. Build lio-utils and install. This is one of the tasks I had difficulties with, for some reason the install scripts were calling out to the incorrect python directory, I found a fix to this myself.
Apply the fix first:
Go into the tcm-py and lio-py directories inside of the lio-utils directory. Open the install.sh in both the tcm-py and lio-py directories and change the “SITE_PACKAGES” string to reflect the following:
SITE_PACKAGES=/usr/lib/python2.6/site-packages
Remember to do this in both the install.sh files for lio-py and tcm-py. Now on to building and installing lio-utils.
Issue the following commands from the lio-utils directory:
make
make install
And you are now done!
Lio-Target and Lio-Utils have no succesfully been installed. As you can see, this was way easier than my previous tutorials, and doesn’t include and rebuilding of kernels, etc… One of the plus’s is that you actually build the kernel modules for the existing CentOS kernel.
One last thing. Start lio-target by issuing the command:
/etc/init.d/target start
And do a ‘dmesg’ to confirm that it started ok!
As always, feel free to post any comments or questions. I’ll do my best to help!
While most of us frequently deploy new ESXi hosts, a question and task not oftenly discussed is how to properly decommission a VMware ESXi host. Some might be surprised to… Read More
This guide will outline the instructions to Disable the VMware Horizon Session Bar. These instructions can be used to disable the Horizon Session Bar (also known as the Horizon Client… Read More
Normally, any VMs that are NVIDIA vGPU enabled have to be manually migrated with manual vMotion if a host is placed in to maintenance mode, to evacuate the host. While… Read More
You may experience GPU issues with the VMware Horizon Indirect Display Driver in your environment when using 3rd party applications which incorrectly utilize the incorrect display adapter. This results with… Read More
Today we're going to cover a powerful little NAS being used with VMware; the Synology DS923+ VMware vSphere Use case and Configuration. This little (but powerful) NAS is perfect for… Read More
Today we'll go over how to install the vSphere vCenter Root Certificate on your client system. Certificates are designed to verify the identity of the systems, software, and/or resources we… Read More
View Comments
Just a side note,
Stress tested this setup for over 12 hours. No freezing or odd kernel panics (as was observed with CentOS 5 and my previous tutorial).
thanks to your website i have lio running without problems on scientific linux 6.1.
only targetcli is not working correctly: i can only access the backstores... any idea ?
this is what i did to get it working:
yum install system-config-network-tui system-config-firewall-tui gcc make patch python-devel kernel-devel git net-snmp-devel epydoc python-setuptools
easy_install simpleparse
easy_install ipaddr
easy_install netifaces
easy_install configobj
cd /usr/src
git clone git://risingtidesystems.com/lio-core-backports.git lio-core-backports.git
git clone git://risingtidesystems.com/lio-utils.git lio-utils.git
cd lio-utils.git/
git checkout --track -b lio-3.5 origin/lio-3.5
cd ../lio-core-backports.git/
make
make install
cd ../lio-utils.git/
edit tcm-py/install.sh and lio-py/install.sh
#SITE_PACKAGES=`python ../get-py-modules-path.py`
SITE_PACKAGES=/usr/lib/python2.6/site-packages
make
make install
cd /usr/src
git clone git://risingtidesystems.com/configshell.git configshell.git
cd configshell.git
python setup.py install
cd ../
git clone git://risingtidesystems.com/rtslib.git rtslib.git
cd rtslib.git
python setup.py install
cd ../
git clone git://risingtidesystems.com/targetcli.git targetcli.git
cd targetcli.git
python setup.py install
chkconfig target on
mkdir -p /var/target/fabric
!! REBOOT !!
Hi Frederik,
Unfortunately I haven't played with targetcli at all. I've never even touched it...
Right now I'm just using lio-utils for configuration and operations... I'll probably start playing with targetcli soon though... I'll keep you posted on how I make out...
Stephen
Hi Stephen
This is an really good article. Althrough I do have some more questions:
1. have you played with multi-pathing with round-robin selection policy using 2 or more NICs on storage and ESXi hosts?
2. have you tried ESXi 5?
3. since you have an working configuration, would you mind posting the full Lio-Target configuration files of your target (where you've successfuly run vMotion, etc.)?
Thanks and keep up the good work!
Kind regards, Marko.
Hey, Frederik,
Can you post your's full Lio-Target configuration files of your target on SL6.1? I'm also interested in deploying test SL6.1 with Lio-Target and would apprechiate some help!
Kind regards, Marko.
Hi Marko,
Thanks, and I'm glad if the article helped at all!
1. I've done no testing with multi-pathing whatsoever. I just don't have the equipment available, and my main target has just been to get Lio-Target running stable and properly to provide VMFS over iSCSI to multiple ESXi hosts in a cluster. Maybe one day in the future, no doubt I will post a blog entry if I do!
2. Unfortunately I haven't tried ESXi 5 yet. My company is a VMware partner, unfortunately we didn't sell enough licensing last year to make the cut for the "Solution Provider" tier. So I'm stuck with 4.x licenses for my testing/training environment. If I do make some sales, and hit the "Solution Provider" tier, upgrading my test/dev environment to 5.x will be the first thing I'll do.
3. I'll post the config below. Keep in mind that I configured Lio at first using Lio-Utils, then used tcm_dump to create the bootup config. I won't post the bootup config (as it's generated by using lio-utils and tcm_dump), but here's what I typed to configure the target.... Keep in mind I'm using a HP SmartArray 6402 controller so that's why you see /dev/cciss/c0d0 instead of /dev/sdb. Also, keep in mind there's no fancy configuration required to be able to use Storage vMotion, or vMotion... Just keep in mind that if you use SCSI pass-through, that your SCSI device itself supports SCSI persistent reservations, if not, make sure you use tcm_loop and blockio which handles reservation emulation... I don't know if the SmartArray 6402 controller and MSA20 arrays support persistent reservations, so I just used blockio to be safe.
Here's what I typed to configure the target:
# First I start the target
/etc/init.d/target start
# Then I configure tcm_loop and use blockio to setup the arrays to be accessible by lio-target. /dev/cciss/c0d0 is a HP MSA20 Storage Array disk array. If this were just a normal scsi disk you'd simply put /dev/sdb or so the like.
tcm_node --block iblock_0/array1 /dev/cciss/c0d0
tcm_node --block iblock_1/array2 /dev/cciss/c0d1
# Now we configure lio, setup the LUNs, setup the Target Portals and setup LUN ACLs for security, and finally enable the portals. I've changed iqns and IPs below for my own privacy. 192.168.0.1 would be the IP of the iSCSI server/target itself.
lio_node --addlun iqn.2010.com.stephenwagner.iscsi:array1 1 0 iscsi00 iblock_0/array1
lio_node --addlun iqn.2010.com.stephenwagner.iscsi:array2 1 0 iscsi01 iblock_1/array2
lio_node --addnp iqn.2010.com.stephenwagner.iscsi:array1 1 192.168.0.1:3260
lio_node --addnp iqn.2010.com.stephenwagner.iscsi:array2 1 192.168.0.1:3260
lio_node --disableauth iqn.2010.com.stephenwagner.iscsi:array1 1
lio_node --disableauth iqn.2010.com.stephenwagner.iscsi:array2 1
lio_node --addlunacl iqn.2010.com.stephenwagner.iscsi:array1 1 iqn.1998-01.com.vmware:esx02 0 0
lio_node --addlunacl iqn.2010.com.stephenwagner.iscsi:array1 1 iqn.1998-01.com.vmware:esxi01 0 0
lio_node --addlunacl iqn.2010.com.stephenwagner.iscsi:array1 1 iqn.1998-01.com.vmware:esx03 0 0
lio_node --addlunacl iqn.2010.com.stephenwagner.iscsi:array2 1 iqn.1998-01.com.vmware:esx02 0 0
lio_node --addlunacl iqn.2010.com.stephenwagner.iscsi:array2 1 iqn.1998-01.com.vmware:esxi01 0 0
lio_node --addlunacl iqn.2010.com.stephenwagner.iscsi:array2 1 iqn.1998-01.com.vmware:esx03 0 0
lio_node --enabletpg iqn.2010.com.stephenwagner.iscsi:array1 1
lio_node --enabletpg iqn.2010.com.stephenwagner.iscsi:array2 1
Let me know if you need anything else :)
Stephen
i'm going to test the multipathing configuration today or tomorrow on ESX 5
i'll provide my findings asap, including documentation
Oh and by the way...
Here's just a performance note:
Lio-Target server is an older Xeon 3.2Ghz Single core with hyperthreading, 512MB of SDRAM
Storage Arrays are 2 X HP MSA20 (1 X RAID 5, 1 X RAID 10)
I've been getting 90-110MB/sec constant network throughput, CPU usage usually sits around 30% at those high speeds.
I REALLY want to set up trunking and see what kind of speeds this bad boy can do!
@Frederik
did you by any chance find a solution for the targetcli problem or are you just using the lio utils as well now? Because i am having the exact same issue atm where i can only access the backstores with it.
Hi there, I was wondering if you have had any chance to test LIO on Ubuntu. I have wanted to set it up for some time and each time I get nowhere.
You had talked about using LIO 4.x when you get it working, I was wondering if you have had a chance to mess with LIO 4.x, I am told it is supposed to be "baked in" to the 3.1.x Linux kernel. Though it does not appear to be in my Ubuntu server. Is it possible it need to be selected at the time you configure the kernel?
Hi Ian,
Originally my iSCSI production box was running Ubuntu with Lio 3.x, not 4.x. When it came time for me to upgrade, I spent a really long time trying to get 4.x running (especially since I thought it was incorporated into the Linux kernel).
It turns out that even though Lio 4.x was incorporated in to the kernel, it was still under development. The kernel that I was using actually did not contain the iSCSI component. I took a look at the Lio-Target website and it looks like it wasn't put in place until 3.1 in November 24th of 2011.
Are you using Kernel 3.1? Or 3.0? This could be one of your issues... I know that distribution kernel releases are usually behind kernel.org releases. Also there is also the possibility that they may not have compiled lio into the standard release kernel (if indeed the kernel release is 3.1.x).
I'm actually still running Lio-Target 3.5 backport on my CentOS 6 box, and absolutely love it. One day I'll give 4.x a try, but it probably won't be until the new year.
Stephen
Thanks for replying, I actually thought that I was running on kernel 3.1, but then found out that I am running on 3.0.0.14. I also saw the same info you saw about the iSCSI component not being added until 3.1, so I went and tried to compile a 3.1 kernel. Got the compile done but still having trouble getting the target to work right. As far as the backports are concerned would someone who using the target for testing really notice a huge difference between 3.5 and the new 4.0 target? Because I might just wait until Ubuntu releases 12.04 that is when they will move to kernel 3.2 supposedly.
Also have you found a serious manual or anything for LIO, something that shows a layout of all of the commands and such?
I'm not too sure of any major differences, but I know there is added support for technologies that weren't included in 3.x. But to be honest, I doubt you using any of these.
Another thing, I could be wrong, but I think lio-utils is being depreciated for Lio 4.x for another configuration application (possibly RTSadmin).
I haven't found a full admin, but here's some of the pages that got me going on configuring Lio-Target:
http://linux-iscsi.org/wiki/Lio-utils
http://linux-iscsi.org/wiki/Lio-utils_HOWTO
Thanks for the info, it was very helpful (your entire blog).
I now have a fully working LIO 4.x target running on Ubuntu server 11.10 64-bit.
I went and compiled my own kernel based on Ubuntu's 12.04 source, this is where I was able to get my hands on a 3.1 kernel so I have iSCSI support.
As far as the tools are concerned as of right now they are not getting rid of "lio-utils" but they are adding a "better" (need a lot of work still) config tool called "targetcli". I have built it and found it a little confusing to work with, mainly when it comes to tweaking the parameters of the target. IMO lio-utils may be a little more complex but its nothing a pen, paper and about 20 mins can't fix.
Now if I can just get lio-utils to install in Python 2.7 correctly I will be all set. (Ubuntu 11.10 uses python2.7 by default)
Well I'm glad you hear you have it up and running!
What type of issues are you having with Python? On CentOS I've had some issues with Lio-Utils and python, but found a work around for it. In my case the issue was placing tcm_node, lio_node, and all the other applications in to the wrong directory. I'm not sure if this is the same issue that is occurring on your side?
Stephen