Jun 302012
 

As most of you know, I have 2 Raspberry Pi.  One has tons of storage, that I do a lot of my hacking on, and development. The other has less storage, I usually keep clean, and don’t do anything funky on it. I like the second to have a clean usable image (even though this doesn’t make sense, and I don’t use it in production, I call the second my “Production Pi”, the first is my “devpi”).

For my development Raspberry Pi, I maintain two installs. One install is a Fedora 17 ARM install with my own kernel that all fits on a 16GB SD card, but my second install (the crazy packed install) actually boots it’s kernel off a small 2GB SD card, and loads it’s rootfs off of a external 500GB USB drive. The SD card install was easy, however the rootfs on USB took a bit of work, I’m going to share with you how I did this.

To have your Raspberry Pi use a rootfs off a USB drive you will need to know what modules your USB drive currently uses on a un-modified Raspberry Pi kernel, and you will need a seperate linux box, and SD card reader/writer to prepare the rootfs and perform these instructions. In my case, I used my 16GB install as a template and cloned it to my 500GB USB drive.

DISCLAIMER: These instructions are what I did to setup a rootfs on a USB device. I am in no way telling you to follow these instructions, and if you do, and any damage occurs I am not liable or responsable. These instructions if not followed properly can cause damage to your existing linux install on your computer and your Raspberry Pi. Only follow these if you understand them.

 

Let’s get started, first let’s prepare the new bootable SD card which loads our kernel:

1) First we want to take a image of the partition table, and boot partition we already boot from on our existing working Raspberry Pi Fedora 17 ARM install. We only want to image the partition table and boot partition. First we use fdisk to find out how big the boot partition is:

fdisk /dev/sdb

Press “p” and hit enter to print the partition table. In my case it outputted:

Welcome to fdisk (util-linux 2.21.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): p

Disk /dev/sdb: 16.0 GB, 16009658368 bytes
64 heads, 32 sectors/track, 15268 cylinders, total 31268864 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *          63     1044224      522081    c  W95 FAT32 (LBA)
/dev/sdb2         1044225    27074558    13015167   83  Linux
/dev/sdb3        27074559    31268863     2097152+  83  Linux

This shows that my boot partition starts at sector 63 and goes till 1044224. Each sector is 512 bytes, for a total boot partition that is 534642688 bytes, or ~509 megabytes. The reason that the partition starts at sector 63, is because everything prior to that on the SD card contains partition information, etc… When we image, we will start from the beginning to grab that partition information. After you record the End sector for your boot partition, hit “q” and enter to exit fdisk.

Let’s create the image. In order to do this, you need to find out what sd* device your SD card is on your current system. If you do a dmesg after connecting the sd card, you should see it. Make sure you choose the correct sd* or you could damage your current linux system. In my case, after I connected the SD card to my linux workstation, it became /dev/sdb.

Let’s make the image:

dd if=/dev/sdb of=image.img bs=512 count=1044224

This will create an image called image.img of the boot partition and partition table. Notice how we use a bs=512 (this is because each sector is 512 bytes), and a count=1044224 (this is the end of the boot partition).

 

2) Let’s copy the SD Card’s rootfs to a external drive! Hook up your empty external hard drive. On this drive we will do a few things. First, partition the drive so we have two partitions (one for the rootfs, and one for swap), then we will make the filesystems, and finally copy over the rootfs to the new drive.

In my case, when I hooked up my external drive to my computer, it was assigned as /dev/sdc, the SD card is still /dev/sdb. Keep in mind this may be different on your computer, if you use the incorrect values, you may actually damage your current linux workstation. You can always use dmesg after hooking up a usb device to find out what sd* it was assigned.

First let’s create a patition table on the usb drive. Again, on my computer the usb drive is sdc, your’s may be different.

fdisk /dev/sdc

In fdisk, we will create a partition for an ext4 filesystem. Press “n” for new partitions, follow the instructions, create a primary partition, and let fdisk choose the start and ending. This will use all the space on the hard drive. When done creating, press “w” and enter to write the partition table. This created the /dev/sdc1 partition. Now we need to create the filesytem and label it.

mkfs.ext4 /dev/sdc1 -L rootfs

This creates a ext4 filesystem and labels it “rootfs”.

 

3) Let’s copy the old rootfs to the new one. First we need to mount both of them, and copy everything over.

To do this, we are going to create two folders which we will mount both of the rootfs too.

mkdir rootsource

mkdir rootdest

Then we will mount them:

mount /dev/sdb2 rootsource/

mount /dev/sdc1 rootdest/

And finally, we will copy over the rootfs and then unmount the mounts

cp -afv rootsource/* rootdest/

umount rootsource/

umount rootdest/

You have now copied over the root filesystem from your SD card, to the external USB drive.

 

4) Now we are done with the original SD card, you can remove it, and put in your new SD card which will be used to boot the kernel.

First we need to write the image that contains the partition table, and boot partition. We do this by writing the image file you created above to the NEW SD card. Again, MAKE SURE YOU KNOW WHAT /dev/sd* device the new card is being recognized as.

dd if=image.img of=/dev/sdb

This will write the image to your new SD card. After this, even though we only took the partition table, and the boot partition, the partition table still contains information of the old rootfs. Let’s clean this up (even though we don’t have to).

fdisk /dev/sdb

Inside of fdisk, we will hit “D” to delete all bogus partitions EXCEPT for the boot partition. In my case I had 2 I had to remove, partition 2 and 3. You will probably only have one, which will be partition 2. After you’re done deleting, hit “w” and enter to save.

 

4) Now we have to go to the boot partition and prepare it to boot off the USB drive.

Let’s create a directory to mount and work inside of, then we will update the cmdline.txt file which the Raspberry Pi uses for information to boot from.

mkdir bootmount

mount /dev/sdb1 bootmount

nano bootmount/cmdline.txt

We are now looking at the cmdline.txt file. We need to update it to boot from the USB drive rootfs partition which we created above. Keep in mind, when you hook the USB drive up to your Raspberry Pi, it will be the only /dev/sd* device, so it will probably appear as /dev/sda if you have no other USB drives connected to it. Update yours, here’s an example of mine:

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/sda1 rootfstype=ext4 rootwait text

Save and close the cmdline.txt and then unmount.

umount bootmount

 

So here is where we sit (in order):

-We imaged the old boot partition

-We prepared the new USB drive

-We copied the rootfs to the new USB drive from the SD card

-We wrote the boot image to the new SD card

-We configured the new SD card to boot using the external drive as a root fs.

 

All we have left to do is to is connect the USB drive to your Raspberry Pi, and insert the new boot SD card in to the Pi as well. Boot your Raspberry Pi, if all is well, it should boot off your new rootfs (on the USB drive). This will improve speed since the USB drive is WAY faster than the SD card, and now you’re SD card will only be used to boot the kernel.

Important Notes:

-If these instructions don’t work, chances are your USB device may require a driver that is not built in your current Raspberry Pi linux kernel, you will need to identify what module your USB device uses by issuing a “lsmod” when you have booted off your old SD card, then re-compiling the kernel with this built in to the kernel and NOT as a module. Instructions on compiling a kernel can be found here: http://www.stephenwagner.com/?p=616

-If you really feel comfortable with Linux, after you do this, you could re-size the partitions, and create a SWAP partition for your Raspberry Pi. ONLY setup a Swap partition if you are using an external USB drive, as SD Card’s are NOT fast enough, and swap’ing can actually limit the life of a SD card due to the I/Os.

Jun 302012
 

A number of projects I’m currently working on and some that I haven’t even started yet, require me to compile my own kernel. The reason I need to do this is to be able to build various code in to a live running kernel, or make various modules available for hardware/applications I need or will need in the future.

 

A few reasons why you may require a custom built kernel:

-Hardware support for particular hardware you are using

-Support for USB drives, USB devices, Wifi, etc…

-Support for kernel features (special networking features, software features, etc…)

-Built support for booting off different rootfs devices (NFS, iSCSI, USB devices)

The list goes on and on, but you have the idea. And chances are if you came to this page, you already have your reason, you’re just trying to figure out how. Well it’s pretty easy on Fedora 17 ARM on your Raspberry Pi. Keep in mind you can use a cross-compiler and build the kernel on your desktop x86 machine and move it over, but I preferred to actually build the kernel on my Raspberry Pi.

Here’s how:

1) First let’s grab all the packages we need. We will issue a yum command to make sure we have the compilers, libraries, and other stuff we will need. We will also install “screen” so that we can hide the terminal session where we are compiling so you don’t need to leave your ssh session open.

yum install gcc gcc-c++ gcc-gfortran glib libtool gtk+ gtk+extra gtk2 git ncurses-devel kernel-headers screen

This will install everything and any dependencies that are required.

 

2) Let’s start a screen session. A screen session is a terminal session you can attach/detach on demand. Think of it as a terminal that you can keep open even in the back ground. You can start it in a ssh session, close your ssh session, and then connect back to the screen session later when you ssh back in.

screen -S kernelcompile

This starts a session called kernelcompile. At any time you can detach the session by pressing Ctrl+A then pressing the “D” button. To re-attach, simply issue “screen -r”.

 

3) Download the kernel sources (with the Raspberry Pi patches built in). You can grab a copy off of the snapshot off the github Raspberry Pi kernel repo.

wget https://github.com/raspberrypi/linux/tarball/rpi-patches
mv rpi-patches kernel.tar.gz

tar zxvf kernel.tar.gz

This will download the sources as a file called rpi-patches, and we just rename it then untar it.

 

4) In my case, it resulted in a folder called “raspberrypi-linux-f679f05”. Here we change directory in to the kernel source, and setup the default kernel config options. And then run “make menuconfig” to bring up the script to allow us to configure our kernel. This is where you choose your options as to what you want to build in to the kernel, and what you want to build as kernel modules that can be loaded after the kernel is booted.

cd raspberrypi-linux-f679f05

cp arch/arm/configs/bcmrpi_defconfig .config

make menuconfig

IMPORTANT: You need to append a version on to the kernel version number, so that you don’t screw your current existing modules up (since there’s a chance if you didn’t edit this, they would be overwritten, or your build would fail when building your modules. You can do this in “make menuconfig” under General Setup. Or you can edit your .config file:

nano .config

Look for “CONFIG_LOCALVERSION”. In my case, mine is set to: CONFIG_LOCALVERSION=”.001″ this will put a .001 at the end of the kernels versions tring, and also after the module version number. Example: /lib/modules/3.1.9.001

After you are done with menuconfig, simply exit out of “make menuconfig” and save your config.

 

5) Now it’s time to start the compiling process. Typing “make” will compile the kernel, and typing “make modules_install” will build the modules and install them to the “/lib/modules” directory.

make

make modules_install

cd..

If the building completed then you have successfully built a linux kernel for the Raspberry Pi and also built/installed the modules that come along with it… We aren’t done yet though. We issue a cd.. to get out of the kernel source directory and back in to the directory that holds it.

 

6) Next we need to download the Raspberry Pi tools which will contain a python application called “imagetool-uncompressed.py”.

wget https://github.com/raspberrypi/tools/zipball/master

mv master tools.zip

unzip tools.zip

We have downloaded the tools, and unzipped them. It might be an idea to issue a “ls” to note the name of the directory it created.

 

7) No we need to enter the tools directory. In my case the directory was called “raspberrypi-tools-772201f” but it will be different for you. After we will build the image.

cd raspberrypi-tools-772201f/

cd mkimage/

python2 imagetool-uncompressed.py /path/to/kernel/source/raspberrypi-linux-f679f05/arch/arm/boot/Image

Note, you will have to also change the directory after the imagetool command to directory where you built your kernel. Once this runs, you will be left with a kernel.img file inside of the directory you are currently in. First we will back up the existing kernel, then copy the new kernel to your /boot/ directory.

IMPORTANT: Once you overwrite your existing kernel, if your kernel build was bad, or you don’t have what’s needed to boot, your kernel MAY not boot. This will cause you to restore your old kernel which I am NOT going in to in this blog post!

mv /boot/kernel.img /boot/kernel.img.bak

cp kernel.img /boot/

Above, we have changed the name of your current kernel and added a .bak to the end. Then we copied the new kernel made to the boot directory.

 

Now you can simply reboot your Raspberry Pi, and BAM, with luck you’ll be running your own kernel.

 

PS. If you ever want to back this kernel up, all you need to do is copy kernel.img (in /boot/) to a safe place, and make a copy of your /lib/modules/kernelversion (in my case /lib/modules/3.1.9.001) directory. If you ever re-image your Pi and want to use the kernel you just built, you can copy the kernel.img and the modules directory to the new image and you are good to go! Might also be worth while to make a backup of your kernel sources and imagetool if you need it again!

Jun 292012
 

As most of you have read, I received 2 X Raspberry Pi the other day. I’ve been actively hacking and working away on these lovely little devices.

An updated post on setting up a Raspberry Pi as an iSCSI Target can be found here!

One of the projects I wanted to do, was get Lio-Target (iSCSI Target) running on the Pi. I know that the Pi doesn’t have gigabit networking, but I thought this would still be an interesting proof of concept. Anyways, I got it running, and I have succesfully connected to a USB storage device which was configured as a iSCSI target on my Pi, from my Windows 7 workstation.

This is a brief overview, I will be providing instructions in detail at a later date. Here’s how I did it:

1) Download Fedora 17 for ARM (build for Raspberry Pi).

2) Put latest Firmware and Kernel from Raspberry Pi github repo on to the boot partition. Resize my 16GB card so I have boot, root, and a 2 GB swap.

3) Download snapshot of Raspberry Pi kernel sources. I built the iSCSI Target as modules (I also threw in some other stuff for future projects but it’s not important right now).

3) Install compilers, libraries, etc for kernel build process.

4) Compile kernel

5) Build Raspberry Pi kernel image using Raspberry Pi image tools on github repo, copy to boot.

6) Boot off new kernel

7) Install Target CLI from yum (this was a nice change from compiling on my own), and then build Lio-Utils (this isn’t mandatory, but I like Lio-utils).

8) Configure target, connect, test.

Here’s a copy/paste of proof I have it running!

[root@fedora-arm lio-utils.git]# uname -a
Linux fedora-arm 3.1.9.001 #1 PREEMPT Thu Jun 28 16:40:46 MDT 2012 armv6l armv6l armv6l GNU/Linux
[root@fedora-arm lio-utils.git]# w
09:12:10 up 34 min,  5 users,  load average: 0.99, 1.51, 1.10
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0    host.digitall 31Dec69 22:38   2.27s  0.02s tail -f /var/log/messages
root     pts/1    host.digitall 31Dec69  0.00s  1.19s  0.05s w
root     pts/2    host.digitall 08:58   13:24  13.89s 13.04s top
root     pts/3    host.digitall 31Dec69  0.00s  0.00s   ?    –
[root@fedora-arm lio-utils.git]# /etc/init.d/target status
[—————————] TCM/ConfigFS Status [—————————-]
\——> iblock_0
HBA Index: 1 plugin: iblock version: v4.1.0-rc1-ml
\——-> array0
Status: ACTIVATED  Execute/Left/Max Queue Depth: 0/128/128  SectorSize: 512  MaxSectors: 240
iBlock device: sdb  UDEV PATH: /dev/sdb
Major: 8 Minor: 16  CLAIMED: IBLOCK
udev_path: /dev/sdb

[—————————] LIO-Target Status [—————————-]
\——> iqn.2003-01.org.linux-iscsi.fedora-arm.armv6l:sn.4682cf8cdeec
\——-> tpgt_1  TargetAlias: LIO Target
TPG Status: ENABLED
TPG Network Portals:
\——-> IP-hidden:3260
TPG Logical Units:
\——-> lun_0/30b42bf9f5 -> target/core/iblock_0/array0

Target Engine Core ConfigFS Infrastructure v4.1.0-rc1-ml on Linux/armv6l on 3.1.9.001
RisingTide Systems Linux-iSCSI Target v4.1.0-rc1
[root@fedora-arm lio-utils.git]# cat /proc/cpuinfo
Processor       : ARMv6-compatible processor rev 7 (v6l)
BogoMIPS        : 697.95
Features        : swp half thumb fastmult vfp edsp java tls
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xb76
CPU revision    : 7

Hardware        : BCM2708
Revision        : 0002
Serial          : 00000000c1ad6318
[root@fedora-arm lio-utils.git]#

Jun 272012
 

Well, it’s official. I am now a happy owner of two Raspberry Pi computers.

I’m busy tinkering away but I just wanted to upload some pictures and videos. I will be creating a separate post outlining what I’m hoping to accomplish with these little devices, so stay tuned!

Scroll to the bottom of this post for videos!

Element 14 shipping package (origina Toronto, Ontario)RS Electronics Box containing Raspberry Pi and 4GB SD Card

Two Raspberry Pi's in their shipping boxes (Left Element14, Right RS Electronics)Two Raspberry Pi on their electro-static wrappers.

Raspberry Pi hooked upRaspberry Pi

Raspberry Pi in operationRaspberry Pi turned On

The workbench for Raspberry Pi development

Jun 092012
 

So there I was… Had a custom built vSphere cluster, 3 hosts, iSCSI target (setup with Lio-Target), everything running fine, smoothly, perfect… I’m done right? NOPE!

Most of us who care, are concerned about our Disaster Recovery or Backup solution. With virtualization things get a little bit interesting in the fact that either you have a CRAZY large setup, and can use the VMware backup stuff, or you have a smaller environment and want something simple, easy to use, and with a low footprint. Configuring a backup and/or disaster recovery solution for a virtualized environment may be difficult and complicated, however after it’s fully implemented; management, use, and administration is easy, and don’t forget about the abilities and features you get with virtualization.

Reasons why virtualization rocks when it comes to backup and disaster recovery:

-Unlike traditional backups you do not have to install a bare OS to run the backup software to restore over

-You can restore to hardware that is not like nor similar to the original hardware

-Backups are now simple files that can be easily moved, transported, copied, and saved on to normal or non-normal media (you could save an entire system on to a USB key if it was big enough). On a 2TB external USB drive, you could have a backup of over 16+ virtual machines!

-Ease of recovery: Copy the backed up VM files to host/datastore, and simply hit play. Restore complete and you’re up and running!

 

So with all that in mind, here we go! (Scroll to bottom of post for a quick conclusion).

For my solution, here are some of the requirements I had:

1) Utilize snapshots to take restorable backups while the VMs are running (no downtime).

2) Move the backups to a different location while running (this could be a drive, NFS export, SMB share, etc…).

3) Have the backups stored somewhere easy to access where I can move them to a removable external USB drive to take off-site. This way, I have fast disk-to-disk access to restore backups in the event my storage system goes down or RAID array is lost (downtime would be minimal), or in the event of something more serious like a fire, I would have the USB drive off-site to restore from. Disk-to-disk backups could happen on a daily basis, and disk-to-USB could be done weekly and taken off-site.

4) In the event of a failure, be able to bring USB drive onsite, transfer VMs back and be up and running in no time.

 

So with this all in mind, I started designing a solution. My existing environment (without backup) composed of:

2 X HP DL360 G5 Servers (running ESXi)

1 X HP ML350 G5 Server (running ESXi)

1 X Super Micro Intel Xeon Server (Running CentOS 6 & Lio-Target backports: providing iSCSI VMFS)

2 X HP MSA20 Storage Units

 

First, I need a way to create snapshots of my 16+ virtual machines. After, I would need to have the snapshots moved to another location (such as a backup server). There is a free script available called ghettoVCB. ghettoVCB is a “Free alternative for backup up VM’s for ESX(i) 3.5, 4.x+ & 5.x” and is available (along with tons of documentation) at: http://communities.vmware.com/docs/DOC-8760. This script is generally ran on the ESXi host, generates a snapshot and clones it to a seperate datastore configured on that host. It does this for all virtual machines named in the VM list you specify, or for all VMs on the host if a specific switch is passed to ghettovcb.sh.

So now that we have the software, we need to have a location setup to back up to. We could either create a new iSCSI target, or we could setup a new Linux server and have a RAID array configured and formatted with ext4 and exported via NFS. This would allow us to have the NFS setup as a datastore on ESXi (so we can backup to the NFS export), and afterwards be able to access the backups natively in Linux to copy/move to a external drive formatted with EXT4.

We configure a new server, running CentOS 6 with enough storage to backup all VMs. We create NFS exports and mount these to all the ESXi hosts. We copy the ghettovcb script to a location on the NFS export so it’s accessible to all hosts easily (without having to update the script on each host individually), and we create lists for each physical host containing the names of the virtual machine it virtualizes. We then edit the ghettovcb.sh file to specify the new destination datastore (the backup datastore) and how we want it to back up.

When executing:

./ghettovcb.sh -f esxserverlist01

It creates the snapshot for each VM in the list for that host, clones it to the destination datastore (which in my setup is the NFS export on the new backup server), then deletes the snapshot when the backup is complete, finally moving on to the next VM and repeating the process until done. The script needs to be ran on all hosts, and list files for VMs have to be created for each host.

We now have a backup server and have done a disk-to-disk backup of our virtual machines. We can now plug in a large external USB drive to the backup server, and simply copy over the backups to it.

 

I do everything manually because I like to confirm everything is done and backed up properly, however you can totally create scripts to automate the whole process. After this we have our new backup solution!

 

Quick recap:

1) Setup a new backup server with enough disk space to back up all VMs. Setup an NFS export and mount it to ESXi hosts.

2) Download and configure the ghettoVCB script. Run the script on each ESXi host to disk-to-disk backup your VMs to your new backup server.

3) Copy the backup files from the backup server to a external USB drive that has enough space. Take off-site.

 

I have had to restore a couple VMs in the past due to a damaged RAID array, and I did so using a backup from above. It worked great! I will create a post on the restore process sometime in the future (for now feel free to look at the ghettoVCB documentation)!

Jun 092012
 

Recently, I’ve started to have some issues with the HP MSA20 units attached to my SAN server at my office. These MSA20 units stored all my Virtual Machines inside of a VMFS filesystem which was presented to my vSphere cluster hosts over iSCSI using Lio-Target. In the last while, these logical drive has just been randomly disappearing, causing my 16+ virtual machines to just halt. This always requires me to shut off the physical hosts, shut off the SAN server, shut off the MSA20s, and bring everything all the way back up. This causes huge amounts of downtime, and it just a pain in the butt…

I decided it was time for me to re-do my storage system. Preferably, I would have purchased a couple HP MSA60s and P800 controllers to hook it up to my SAN server, but unfortunately right now it’s not in the budget.

A few years ago, I started using software RAID. In the past I was absolutely scared of it, thought it was complete crap, and would never have touched it, but my opinion drastically changed after playing with it, and regularly using it. While I still recommend businesses to use Hardware based RAID systems, especially for mission critical applications, I felt I could try out software RAID for the above situation since it’s more of a “hobby” setup.

I read that most storage enthusiasts use either the Super Micro AOC-SASLP-MV8, or the LSI SAS 9211-8i. Both are based off different chipsets (both of which are widely used in other well known cards), and both have their own pro’s and con’s.

During my research, I noticed a lot of people who run Windows Home Server were utilizing the AOC Super Micro Card. And while using WHS, most reported no issues whatsoever, however it was a different story when reading posts/blog articles from people using Linux. I don’t know how accurate this was, but apperently a lot of people had issues with this card under heavy load, and some just couldn’t get it running inside of linux.

Then there is the LSA 9211-8i (which is the same as the extremely popular IBM M1015). This bad boy supports basic RAID operations (1, 0, 10), but most people use it with JBOD and simply use Linux MD Software RAID. While there was numerous complaints about users having issues with their systems even detecting their card, other users also reported issues caused by the BIOS of this card (too much memory for the system to boot). When people did get this card working though, I read of mostly NO issues under Linux. Spent a few days confirming what I already had read and finally decided to make the purchase.

Both cards support SAS/SATA, however the LSI card supports 6Gb/sec SAS/SATA. Both also have 2 internal SFF8087 Mini-SAS connectors to hook up a total of 8 drives directly, or more using an SAS expander. The LSI card uses a PCIe (V.2) 8x slot, vs the AOC-SASLP which uses PCIe (V.1) 4x slot.

I went to NCIX.com and ordered the LSI 9211-8i along with 2 breakout cables (Card Part#: LSI00194, Cable Part#: CBL-SFF8087OCF-06M). This would allow me to hook up a total of 8 drives (even though I only plan to use 5). I already have an old computer I already use with an eSATA connector to a Sans Digital SATA Expander for NFS, etc… that I plan on installing the card in to. I also have an old Startech SATABAY5BK enclosure which will hold the drives and connect to the controller. Finished case:

Server with disk enclosures (StarTech SATABAY5BK)

(At this point I have the enclosure installed along with 5 X 1TB Seagate 7200.12 Barracuda drives)

Finally the controller showed up from NCIX:

LSI SAS 9211-8i

I popped this card in the computer (which unfortunately only had PCIe V1), and connected the cables! This is when I ran in to a few issues…

-If no drives were connected, the system would boot and I could succesfully boot to CentOS 6.

-If at all I pressed CTRL+C to get in to the cards interface, the system would freeze during BIOS POST.

-If any drives were connected and detected by the cards BIOS, the system would freeze during BIOS POST.

I went ahead and booted in to CentOS 6. Downloaded the updated firmware and BIOS and flashed the card. The flashing manual was insane, but had to read it all to make sure I didn’t break anything. First I updated both the firmware and BIOS (which went ok), however I couldn’t convert the card from IR firmware to IT firmware due to errors. I google’d this and came up with a bunch of articles, but this one: http://brycv.com/blog/2012/flashing-it-firmware-to-lsi-sas9211-8i/ was the only one that helped and pointed me in the right direction. Essentially just stating you have to use the DOS flasher, erase the card (MAKING SURE NOT TO REBOOT OR YOU’D BRICK IT), and then flashing the IT Firmware. This worked for me, check out his post! Thanks Bryan!

Anyways, after updating the card and converting it to the IT firmware. I still had the BIOS issue. I tried the card in another system, and still had a bunch of issues. I finally removed 1 of 2 video cards and populated the card in a Video Card slot, and I finally could get in to the BIOS. First I enabled staggered spin-up (to make sure I don’t blow the PSU on the computer with a bunch of drives starting up at once), changed some other settings to optimize, and finally disabled the boot BIOS, and changed the option for the adapter to be disabled for boot, and only available to the OS. When removing the card, and putting it in the target computer, this worked. Also noticed that the staggered spin-up started during the Linux kernel startup when initializing the card. Here’s a copy of the kernel log:

mpt2sas version 08.101.00.00 loaded
mpt2sas 0000:06:00.0: PCI INT A -> Link[LNKB] -> GSI 18 (level, low) -> IRQ 18
mpt2sas 0000:06:00.0: setting latency timer to 64
mpt2sas0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (3925416 kB)
mpt2sas 0000:06:00.0: irq 24 for MSI/MSI-X
mpt2sas0: PCI-MSI-X enabled: IRQ 24
mpt2sas0: iomem(0x00000000dfffc000), mapped(0xffffc900110f0000), size(16384)
mpt2sas0: ioport(0x000000000000e000), size(256)
mpt2sas0: sending message unit reset !!
mpt2sas0: message unit reset: SUCCESS
mpt2sas0: Allocated physical memory: size(7441 kB)
mpt2sas0: Current Controller Queue Depth(3305), Max Controller Queue Depth(3432)
mpt2sas0: Scatter Gather Elements per IO(128)
mpt2sas0: LSISAS2008: FWVersion(13.00.57.00), ChipRevision(0x03), BiosVersion(07.25.00.00)
mpt2sas0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
mpt2sas0: sending port enable !!
mpt2sas0: host_add: handle(0x0001), sas_addr(0x5000000080000000), phys(8)
mpt2sas0: port enable: SUCCESS

SUCCESS! Lot’s of SUCCESS! Just the way I like it! Haha, card intialized, had access to drives, etc…

Configured the RAID 5 Array using a 256kb chunk size. I also changed the “stripe_cache_size” to 2048 (the system has 4GB of RAM) to increase the RAID 5 performance.

cd /sys/block/md0/md/

echo 2048 > stripe_cache_size

At this point I simply formatted the drive using EXT4. Configured some folders, NFS exports, and then used Storage vMotion to migrate the Virtual Machines from the iSCSI target, to the new RAID5 array (currently using NFS). The main priority right now was to get the VMs off the MSA20 so I could at least create a backup after they have been moved. Next step, I’ll be re-doing the RAID5 array, configuring the md0 device as a iSCSI target using Lio-Target, and formatting it with VMFS. The performance of this Software RAID5 array is already blowing the MSA20 out of the water!

So there you have it! Feel free to post a comment if you have any questions or need any specifics. This setup is rocking away now under high I/O with absolutely no problems whatsoever. I think I may go purchase another 1-2 of these cards!

Jun 062012
 

So, as most of you know, I have TONS of articles pertaining to getting Lio-Target running on Cent OS. In the beginning, things seemed rather “hit or miss” due to weird errors when either building lio-target or lio-utils…

Turns out, most of the issues I’ve had are related to Python and the current running version. Recently I updated one of my storage boxes using yum, and it completely broke Lio-Target, and Lio-Utils when I had to rebuild them for the new kernel. I was in a panic to mount an old CentOS 6 ISO to get one of the first Python version for CentOS. After downgrading, I was able to build and install both.

 

Just a heads up for you people getting weird python errors.

Jun 032012
 

Well, for the longest time I have been running a vSphere 4.x cluster (1 X ML350 G5, 2 X DL360 G5) off of a pair of HP MSA20’s connected to a SuperMicro server running Lio-Target as a iSCSI Target on CentOS 6.

This configuration has worked perfectly for me for almost a year, requiring absolutely no maintenance/work at all.

Recently I moved, so I had to move all my servers, storage units, etc… When I got in to the new place, and went to power everything up, I noticed that my first drive had failed upon initializing one of the MSA20 units.. I replaced this drive, let it rebuild, and thought this would be the end of the issue, but I was incorrect. (Just so everyone knows, these units had been on continuously for 8+ months before turning them off to move).

For months since this happened and a successful rebuild, at times of high I/O (backing up to a NFS share using GhettoVCB), the logical drive in the array just disappears. I have each MSA20 connected to it’s own HP SmartArray 6400/6402 controller. When the logical drive disappears, I notice that the “Drive Failure” LED on the SA 640x controller illuminates. When this happens, I have to shut off all physical servers, the storage server (running Lio-Target), and the MSA’s, and restart everything.

Sometimes it is worse than others (example I’ve been dealing with this issue non-stop all weekend with no sleep). Even under low I/O I’ll be starting the VMs and it will just lose it again. While other times, I can run it for weeks as long as I keep the I/O to a minimum.

I’ve read numerous other articles and posts of other people having the same issue. These people have had HP replace every single item inside of the MSA20 unit (except the drives) and the issue still occurs. This made me start thinking.

This weekend, I NEEDED to get a backup done. While doing this, the issue occurred, and it got to the point where I couldn’t even start the VMs, let alone back them up. I figured that since other people have had this issue, and since replacing all hardware hasn’t fixed it (I even moved a RAID array from one MSA20 to the other I have with no affect), then it has to be the drives themselves.

There are two possibilities, either the drives are failed, and the MSA20 isn’t reporting them as failed (which I’ve seen happen), or the way the MSA20 creates the RAID array has issues. I ran a ADU report and carefully read the entire report. Absolutely no issues reading/writing to the drives, and the MSA20 has no events in it’s log. It HAS to be the way the RAID array is created on the disks.

Please do not try this unless you know exactly what you’re doing. This is very dangerous and you can lose all your data. This is just me reporting on my experience and usage.

In desperation I thought to myself this all happened when a drive failed and I put a new disk in the array. Since I couldn’t back any of my data up, or let alone even start the VMs, I decided to start behaving dangerously. I kept all my vSphere hosts offline, and just turned on the MSA20 units and the SuperMicro server that is attached to them. I then proceeded to remove a drive, re-insert it, let it rebuild over 3 hours, then when healthy and rebuilt, do it to the next drive. I have a RAID 5 Array containing 4 X 500GB disks, so this actually took me a day to do (had to remove/re-insert, rebuild, then next drive).

After finally removing and rebuilding each drive in the array, I finally decided to boot up the vSphere servers, and run a backup of my 12 VMs. Not only did everything seem faster, but I completed the backup without any problems. This shows that there’s a very high chance that the RAID stuff on the drives is either corrupt, damaged, or just wasn’t implemented very nicely. Rebuilding each drive seemed to fix this. I’ll report in a few weeks to let you know for sure that it’s resolved!

 

Hope this helps someone out there!