PLEASE VISIT http://www.stephenwagner.com/?p=300 FOR AN UPDATED TUTORIAL.
Disclaimer: Please note that whenever doing any of the steps mentioned in this post, if you do not know what you are doing, you can render your linux install usless. Please do NOT use this in a production environment, and only use for testing. I’m not liable if you toast your linux install.
First and foremost, you need to know that from what I understand, the typical kernels that ship with CentOS have numerous patches, modifications, and updates from the typical Linux kernel releases. This is one reason why the releases of the kernels are always behind compared to ones that are actively being developed. I could be wrong, but I think the Redhat kernel patches are applied to CentOS kernels.
This tutorial will walk you through on how to get you up and running in the dirtiest and quickest method. You may and probably will have to modify things, re-compile, etc… to resolve any issues you may run in to.
I wrote this article only to help more people get up and running with one of the only open-source Linux iSCSI targets that has been certified by VMware (Certified when running on certain appliances) to be used in a vSphere environment (supports Vmotion, SCSI reservations, etc…)
I’m going to assume you have CentOS 5.5 installed and is fully up to date.
Anyways, here’s a breakdown of what we will be doing to get Lio-Target to run on CentOS:
- Download lio-target modified kernel
- Download lio-utils
- Compile a modified kernel using existing kernel .config file
- Compile and install lio-utils
- Sample commands to setup a dedicated drive on your system as a iSCSI target (to test)
- Compose two quick and dirty config files so lio-target can run
Download the lio-target modified kernel
For this step, you will need to have git installed. From what I understand git is not an option during the CentOS install, and cannot be installed using the default typical yum repos. To get git installed, we will first add the “RPMFusion” and “EPEL” yum repos.
(Info on how to install these can be found at http://rpmfusion.org/ and https://fedoraproject.org/wiki/EPEL)
Once you have installed these, it’s time to install git. This can be easily done by typing:
yum install git
After git is installed, it’s now time to download the lio-target kernel.
Type in:
git clone git://git.kernel.org/pub/scm/linux/kernel/git/nab/lio-core-2.6.git lio-core-2.6.git
It is now downloading the source. After this is complete, change directory to the new directory that was created, and type in:
git checkout origin/lio-3.4
You now have the source.
Download lio-utils
This is a pretty simple step. If you just completed the step above, make sure you cd back in to a directory that you can use as a workspace, and make sure you do NOT do this, inside of the directory that was downloaded above. Type in:
git clone git://git.kernel.org/pub/scm/linux/storage/lio/lio-utils.git lio-utils.git
This should create a directory called “lio-utils.git”. This step is done.
At this point
You should have two directories inside of your workspace:
lio-core-2.6.git
lio-utils.git
Compile a modified kernel using existing kernel .config file
It is now time to build a kernel. First of all, issue a uname –a and note the kernel you are running. In my case it’s “2.6.18-194.17.1.el5”. I’m going to take a config file from the boot directory that matches the number, and copy it to the lio-core-2.6.git directory, only change the name to “.config”. This is what I would type in on my end:
cp /boot/config-2.6.18-194.17.1.el5 /root/lio-core-2.6.git/.config
This command above would copy the config for the CentOS kernel, and move it to the lio-core-2.6.git directory with the new name of “.config”. Now change into the lio-core-2.6.git directory:
cd lio-core-2.6.git
At this point we are going to run a command to help the config adjust to the newer kernel version. Type in:
make oldconfig
This will spawn numerous messages. You can just go ahead and keep hitting enter. It will seem like an endless loop, however eventually it will complete. Now the next step, this is important. By default, new kernels use a different sysfs structure. We need to turn on the depreciated item that CentOS uses. Type in:
make menuconfig
Navigate to:
“General Setup”, then check (put a star in the box) for “enable deprecated sysfs features to support old userspace tools”.
Now hit tab to select exit. Once again tab to exit at the main menu. We are now ready to compile! To do a quick compile, type in:
make && make modules && make modules_install && make install
Feel free to disapeer for a while. This will take some time depending on the performance of your system. Once done, you now have a kernel with lio-target built in that is compiled and installed.
Keep in mind that this is NOT the default kernel, and you will have to select this to boot when starting your system. To change this, modify /etc/grub.conf and change the default value to whichever item it is (remember that the first item is 0, and not 1).
Let’s boot the new kernel. Safely shut down your linux box, reboot, and when grub shows up, boot using the new kernel you compiled.
Please note: This guide is a quick and dirty way to get this up and running. Since we skipped customizing the kernel, the kernel you compiled will no doubt come up with errors on boot. I simply ignore this. You can too, just to get this up and running. You can come back at a later time and refine your custom kernel to be used.
Compile and install lio-utils
This is simple. Change directory to the directory where you downloaded lio-utils.
cd /root/lio-utils.git
And to compile and install:
make
make install
And BAM, you’re done, that was easy!
Sample commands to setup a dedicated drive on your system as a iSCSI target (to test)
It’s show time. I’m still fairly new to the configuration and usage of lio-target, so I’m just posting the commands to get it working on a dedicated disk. There are no manual (man page) entries installed by lio-utils, so you will need to use lio_node –help and tcm_node –help for more information on proper usage.
Keep in mind, DO NOT use your linux disk as a iSCSI target! Let’s pertend we have a second disk in the system /dev/sdb that we want to turn in to a target. This is what we would type:
tcm_node –block iblock_0/array /dev/sdb
/etc/init.d/target start
lio_node –addlun iqn.2010.com.stephenwagner.iscsi:array 1 0 iscsi00 iblock_0/array
lio_node –listendpoints
lio_node –addnp iqn.2010.com.stephenwagner.iscsi:array 1 192.168.0.10:3260
lio_node –listendpoints
lio_node –disableauth iqn.2010.com.stephenwagner.iscsi:array 1
lio_node –addlunacl iqn.2010.com.stephenwagner.iscsi:array 1 iqn.CLIENTINITIATORHERE.com 0 0
lio_node –enabletpg iqn.2010.com.stephenwagner.iscsi:array 1
The above will create a target, and discovery portal for /dev/sdb on 192.168.0.10. This will also disable CHAP authentication, and will allow the initator I specified above to connect.
Please change /dev/sdb to the drive you want to use
Please change iqn.2010.com.stephenwagner.iscsi:array to the iSCSI target you want to call yours.
Please change 192.168.0.10 to the IP of the iSCSI target your configuring. Leave port as 3260.
Please change iqn.CLIENTINITIATORHERE.com to the iqn for your initiator (client). This will be set on the client you are using to connect to the iSCSI target.
BAM, your target is up and running! Keep in mind, this configuration is lost upon reboot.
Compose two quick and dirty config files so lio-target can run
Here’s what you need to put in the config files to make the above config work on boot:
/etc/target/tcm_start.sh should contain:
tcm_node –block iblock_0/array /dev/sdb
/etc/target/lio_start.sh should contain:
lio_node –addlun iqn.2010.com.stephenwagner.iscsi:array 1 0 iscsi00 iblock_0/array
lio_node –addnp iqn.2010.com.stephenwagner.iscsi:array 1 192.168.0.10:3260
lio_node –disableauth iqn.2010.com.stephenwagner.iscsi:array 1
lio_node –addlunacl iqn.2010.com.stephenwagner.iscsi:array 1 iqn.1991-05.com.CLIENTINITATOR.com 0 0
lio_node –enabletpg iqn.2010.com.stephenwagner.iscsi:array 1
After you make these config files. You should be able to start lio-target in a running state, by issuing:
/etc/init.d/target start
And remember, that you can always view a live feed of what’s going on by issuing:
tail –f /var/log/messages
Hope this helps!
Hi Stephen,
Thanks it is Great information.
This information will help me.
What you suggest if i want to create cluster SAN.
LIO-Target+DRBD+heartbeat make sense?
Glad to hear it helped!
As I get more and more into using Lio-Target, I did come across DRBD. It looks like they would be a perfect match for each other. Although I personally have not touched DRBD yet. (I plan on it though).
Keep in mind as of right now I’ve been using all this in test environments, however I see it very promising, and I’m actually at the point where I’m about to start building a storage system for use in my own production environment.
I know that RisingTide System (http://www.risingtidesystems.com) who is the creator of the Lio-Target have their own cluster storage system so it may be worth checking out there site. I’ve actually been in touch with them and it looks like they are actually doing some pretty cool stuff with the clustering, storage, etc…
What are you using this for in your environment if you don’t mind me asking?
Stephen
Thank for quick response.
Currently i have configured Openfiler in one of our office branch. But problem with openfiler SAN is, windows 2008 doesn’t support SCSI-3 Persistent Reservations. So i did some searching and find LIO-Target suitable for this.
I am also trying to configure LIO-Target with DRBD+heartbeat in my test environment(VM) with high availability. I will move setup on LIO if goes successful.
sem
Yes, that’s one of the big things that scare me about all the open source iSCSI target solutions.
As of right now, as far as I know, Lio-Target is the ONLY Open Source iSCSI target that actually handles SCSI reservations properly. That’s one of the main reason’s why I’m interested in it so much. It’s perfect for both VMware environments, and also Server 2008 Cluster environments.
What distribution are you trying Lio-Target on? Also, let me know how your progress is on your testing. I’m curious how it works out for you. Also curious in how well it works with DRBD. Keep me posted!
Stephen
I am not sure but SCST also support RESERVE/RELEASE functionality. Take a look (http://scst.sourceforge.net/) for full features
Currently I am using Debian Lenny in my test lab. Now i will never use centos after reading your comment here 🙂 http://www.stephenwagner.com/?p=247
I have some other admin related work but, i will surly update you one this process as i am also intrastate in it.
sem
I am also waiting your post on Lio_Target on Ubuntu 10.10
sem
I heard that SCST does, however from all the research I did over a period of a couple months, it looks like Lio-Target was the best option IMO for a production environment.
I’m a HUGE fan of CentOS (since it’s derived from RHEL), and I still want to try to get Lio-Target running on it. Keep in mind that CentOS and RHEL are behind on kernel releases due to the fact they could be considered Enterprise releases and they also heavily modify their own kernels. They are behind on kernel releases compared to other linux distributions. It’s general practice. Once RHEL 6 comes out, and the new version of CentOS is released I’m sure that Lio-Target will run perfectly on them.
Another thing to consider to, is that Lio-Target is becoming part of the normal kernel source code, so future releases of the linux kernel will ship with Lio-Target. I’m not sure if it’s going to be built in the kernels that ship with CentOS, RHEL, and other enterprise distros (such as SuSE, etc…), but recompiling your own kernel should be easy, and I’m sure Lio will run just fine on them.
As soon as I get some time I’ll be posting a config/review of Lio running on Ubuntu. All I can say now is it’s running PERFECTLY under heavy load (ESXi virtual machines, databases, etc…).
I just don’t feel safe running Ubuntu in a production environment. I’d prefer CentOS.
My intention is not to hurt you. After reading your note, i have deside to focus on one stable OS.
I was started my Linux career with RHEL 4 and I am also working on some RHEL and CentOS.
Sorry if i hurt you
Oh no, not at all 🙂
It’s just my mission to get it running on CentOS, lol. I really want to get it working on Cent OS 🙂
Hello,
I have compile Lio-target as this guide but when i am trying to add scsi target i am facing error
# tcm_node –block iblock_0/my_block0 /dev/sdb
ConfigFS HBA: iblock_0
ConfigFS Device Alias: my_block0
Device Params [‘/dev/sdb’]
sh: line 0: echo: write error: Invalid argument
IBLOCK: createvirtdev failed for enable_opt with echo 1 > /sys/kernel/config/target/core/iblock_0/my_block0/enable
Unable to register TCM/ConfigFS storage object: /sys/kernel/config/target/core/iblock_0/my_block0
What i miss please me…
Hi Babu,
I ran in to the same issue when using the latest version of lio-utils… Try using 3.0 instead of 3.1.
First remove 3.1 by cd’ing into the lio-utils directory, and type “make deinstall” (without quotations). Then remove the lio-utils directory.
To download 3.0 instead, use:
git clone git://git.kernel.org/pub/scm/linux/storage/lio/lio-utils.git lio-utils.git
then cd into the directory and type:
git checkout origin/lio-3.0
This will download 3.0 and it should work. You’ll still need to compile and install them.
Keep in mind that I’m not sure if “make de-install” will work. You may have to take a look at the Makefile and find out what files it installed on your system and manually remove them, before downloading, compiling, and installing 3.0 of the lio-utils.
Hope this helps.
Stephen
hi Stephen,
thanks your replay
Now it’s working…
thanks
after recompiling the kernel i have no new options in grub? do i have to manually add the entry? Whats the entry name?
Thanks!
Are you doing this on CentOS or a different distrobution?
On CentOS, running the “make install” while compiling the kernel should add it to grub…
running through this on Ubuntu 10.10
Hi footastic,
On Ubuntu you have to issue some extra commands to get it to put it in grub after compiling and installing the kernel.
Ubuntu 10.10 has a never version of grub…
Log on as “root”, and issue “update grub” (without quotations). This will scan for kernels and initial ramdisks and add it to the bootloader.
Let me know if this works.
I did figure that bit out – I see additional options – but they both hang at kernel panic unable to mount root fs…
is Lio-target included in any upcoming kernels?
im re-compiling with more modules maybe something was excluded on the fs front… *crossing fingers*
Did you pull your original .config from a kernel that’s running and good already on the system?
yes and i did a make oldconfig
Hmmm..
Just to confirm, you did:
1. cp old config
2. make oldconfig
2. make
3. make modules
4. make modules_install
5. make install
?
yup.. do i need to do make menuconfig and check that option if i am not running centos?
If you’re running Ubuntu, you do NOT need to go into “make menuconfig” and check the devfs thing. That is specific for CentOS only. 🙂
just rebuilt no luck! would the devfs thing make a differnce on ubuntu if it is checked… I think this last rebuild i did check it
I’m not really too sure if it would or wouldn’t. Most of my experience and knowledge is on Redhat based Linux distros, which for the most part require the depreciated devfs thingy…
You may have to rebuild a kernel… Keep in mind if your starting from scratch, redownload the source or, run a “make clean” which will clean the kernel source up… Make sure you give it a new name though before you start working away…
well no luck…. reconfiguring to a ext2 / non LVM
Footastic, WAIT!!!
lol, this JUST came to mind… When I setup my prod environment, I remember that when building a kernel on Ubuntu, that I had to use a special command to make an initial ramdisk.
The normal commands listed above (for CentOS), would work for CentOS, but wouldn’t make a valid initial ramdisk on Ubuntu… This is probably what is screwing up your boot…
Run this:
update-initramfs -c -k 2.6.34-iscsi
swap out 2.6.34-iscsi with your kernel version/name….
I’m soo sorry I didn’t remember this till now…
Afterwards, run the grub command to re-generate the bootlist, and you should be good!
Cool! Trying it now!
ok looks good. and booted!. just have to work out some tcm stuff… !!
OSError: [Errno 2] No such file or directory: ‘/sys/kernel/config/target/core/iblock_0’
Blha! lol
Footastic,
Check out comment number 11 on this blog post. It’ll get you running. I had the same issue, you need to install an older version.
tcm_node –block iblock_0/array /dev/store/test
returns the err no 2 …. weird… even with 3.0…
Don’t forget to start lio (load modules):
/etc/init.d/target start
hmm.. it loads fine.. however theres no /sys/kernel/config/target directory (nothing inside of config)
strange…. maybe all these rebuilds fubar’ed something
You didn’t, don’t worry.
Just follow comment 11 closely. You need to look at the Makefile for utils, and remove the files from the newest version on your system. You can’t run make deinstall either, you have to manually do it.
Once you do this, and build the older version, you will get odd errors when running the utils commands, but they will work…
Went through mkfile.. removed everything that stood out
remaked / installed
still getting errors =\
csi:array 1 0 iscsi00 iblock_0/array
mkdir: cannot create directory `/sys/kernel/config/target’: No such file or directory
Unable to create iSCSI Target Logical Unit ConfigFS directory
Your going to have to play with it… There is a weird combination to getting it going…
Just keep in mind, everytime you restart the system, you have to issue a /etc/init.d/target start after booting.
Footastic,
I’m trying 2.6.38-rc4, tried with numerous builds of lio-utils, and I’m getting errors on /sysfs….
I’ve spent the weekend muckin with it, with no success. I’ll figure it out and post here, I think it’s something messy with CentOS
Hello,
I see that Risingstar Systems offers LIO 3.x backports for older kernels. There is also HOWTO that explain how it should be compiled, without need to recompile whole kernel (http://www.linux-iscsi.org/wiki/Backports_HOWTO).
Did you try these backports on CentOS5.5?
It looked promising, since these backports only add modules which are changed, like lio-target, configfs, etc. However, when I tried to compile, got following errors:
/files/lio-core-backports.git/kernel/drivers/target/target_core_iblock.c: In function ‘iblock_map_task_SG’:
/files/lio-core-backports.git/kernel/drivers/target/target_core_iblock.c:937: error: implicit declaration of function ‘for_each_sg’
/files/lio-core-backports.git/kernel/drivers/target/target_core_iblock.c:937: error: expected ‘;’ before ‘{’ token
Did you try it out?
Thanks,
Nik
Morning Nik,
I have read that the backports work, however I’m still a little bit lost regarding the whole concept. With me, I need all the persistent reservation stuff for my VMware environment to work, and I’m not fully sure if the back ports actually have all the functionality and bugfixes that the latest releases have.
With all my knowledge and skills, I’m almost embarrassed to say that I don’t know what the backports actually contain (I don’t know if I should know this or not, lol).
Does anyone else who posts on here know? Please post and let us know!
Hi
I’m testing lio with scentific linux 6 (rhel6)
I’m testing the backports method.
The modules compile with no problem.
But I have problems with lio-utils which seems incompatible.
I had also reverted from 3.1 to 3.0 because of the same problem.
but now, there are error with the list-endpoints.
Investigating shows that the kernel module interface have changed a bit.
definitely not simple to have iscsi working.
I finally used lio head to make backports working on scientific linux 6.
I had to make some small fixes in the python code to detect scientific linux 6 distro.
It now works with good performance (tested with ext4 over simple gigabit network, raid0 lvm exported):
Version 1.03e ——Sequential Output—— –Sequential Input- –Random-
-Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
re.mnd.fr 7656M 52534 96 105963 25 32554 5 51029 86 109373 6 505.3 1
——Sequential Create—— ——–Random Create——–
-Create– –Read— -Delete– -Create– –Read— -Delete–
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
re.mnd.fr,7656M,52534,96,105963,25,32554,5,51029,86,109373,6,505.3,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
I will now test gfs2 over this.
Thanks for your howto.
Hi stephen,
I tried builiding a kernel on ubuntu10.10 64bit, using following commands:
cp old config
make oldconfig
make
make modules
make modules_install
make install
update-initramfs -c -k 2.6.32-server
update-grub
And after a reboot I get kernel panic with following error.Please help.
[0.658820] pci 0000:00:18.2: no compatible bridge window for [io 0Xf000-0Xffff]
[1.572086] kernel panic – not syncing: VFS: Unable to mount root fs on unkown-block(0,0)
Regards,
Avinash
Hi Stephen,
I am using debian lenny, i upgraded the kernel to 2.6.39.2(which has lio inbuilt )
The debain booted with the new 2.6.39.2 kernel without any problem.
But I have problems with lio-utils,
I first installed lio-utils 3.1, then uninstalled, and install 3.0.
I am not sure whether i deleted all the files while uninstalling 3.1(please let me know a way to proper uninstall)
With lio-utils 3.0 i am getting the following errors
soho-dev-d1:/home/tcm/lio-utils.git# /etc/init.d/target start
Loading target_core_mod/ConfigFS core: [FAILED]: 1
soho-dev-d1:/home/tcm/lio-utils.git# tcm_node –block iblock_0/my_block0 /dev/sdb
ConfigFS HBA: iblock_0
Traceback (most recent call last):
File “/usr/sbin/tcm_node”, line 794, in
main()
File “/usr/sbin/tcm_node”, line 786, in main
(options, args) = parser.parse_args()
File “/usr/lib/python2.5/optparse.py”, line 1385, in parse_args
stop = self._process_args(largs, rargs, values)
File “/usr/lib/python2.5/optparse.py”, line 1425, in _process_args
self._process_long_opt(rargs, values)
File “/usr/lib/python2.5/optparse.py”, line 1500, in _process_long_opt
option.process(opt, value, values, self)
File “/usr/lib/python2.5/optparse.py”, line 782, in process
self.action, self.dest, opt, value, values, parser)
File “/usr/lib/python2.5/optparse.py”, line 802, in take_action
self.callback(self, opt, value, parser, *args, **kwargs)
File “/usr/sbin/tcm_node”, line 252, in tcm_create_iblock
tcm_createvirtdev(option, opt_str, value, parser)
File “/usr/sbin/tcm_node”, line 148, in tcm_createvirtdev
ret = os.mkdir(cfs_hba_path)
OSError: [Errno 2] No such file or directory: ‘/sys/kernel/config/target/core/iblock_0’
Regads,
Raju
Hi,
somehow the problem sloved after installing lio-utils v 4.0 . but I again stuck after few steps(while creating TPG)
Below is what I have done:
soho-dev-d1:/# /etc/init.d/target start
Loading target_core_mod/ConfigFS core: [OK]
Calling ConfigFS script /etc/target/tcm_start.sh for target_core_mod: [OK]
Calling ConfigFS script /etc/target/lio_start.sh for iscsi_target_mod: [OK]
soho-dev-d1:/# tcm_node –block iblock_0/my_iblock /dev/vg_1/lv_1
Status: DEACTIVATED Execute/Left/Max Queue Depth: 0/32/32 SectorSize: 512 MaxSectors: 1024
iBlock device: dm-0 UDEV PATH: /dev/vg_1/lv_1
Major: 252 Minor: 0 CLAIMED: IBLOCK
soho-dev-d1:/# lio_node –addlun iqn.2011.com.raju.iscsi:my_iblock 1 0 my_prtl iblock_0/my_iblock
mkdir: cannot create directory `/sys/kernel/config/target/iscsi’: Invalid argument
Unable to create iSCSI Target Logical Unit ConfigFS directory
I cant able to proceed from here, it seems like there is problem in creating iscsi directory, may be issue in lio_node utility.
The details of my setup can be seen below:
soho-dev-d1:/home/LIO/lio-utils.git# ./autoconfig
ARCH?=i386
AUTO_CFLAGS?= -Dscsi_execute_async_address -DPYX_ISCSI_VENDOR='”Linux-iSCSI.org”‘ -DIQN_PREFIX='”iqn.2003-01.org.linux-iscsi”‘ -DLINUX -DLINUX_SCATTERLIST_HAS_PAGE -DSVN_VSN=\”4.0\”
BASENAME?=.i386
DISTRO?=DEBIAN
KERNEL?=26
KERNEL_DIR?=/lib/modules/2.6.39-020639rc4-generic/build
KERNEL_INCLUDE_DIR?=/lib/modules/2.6.39-020639rc4-generic/build/include
KERNEL_SOURCE_DIR?=/lib/modules/2.6.39-020639rc4-generic/build
KERNEL_VERSION_INFO?=LINUX_KERNEL_26
LIBL?=ia32
LIO_UTILS_REV?=4bd008b1
LIO_UTILS_VERSION?=4.0
LIO_VERSION?=4.0
OSTYPE?=LINUX
RELEASE?=2.6.39-020639rc4-generic
RELEASES?=ARRAY(0x8d3d360)
RPM_DIR?=/usr/src/redhat
Please somebody give some suggestion.
Thanks,
raju
Ok everyone…
I’ve done up a new tutorial, which should avoid a lot of the issues some of you are encountering.
Visit: http://www.stephenwagner.com/?p=300 to view the new tutorial!
Stephen
Hi ,
I want to set up a LIO target on Debian wheeze without targetcli. Please help me in knowing the command set for adding, deleting and modifying LIO targets.