My beginning steps in iSCSI

Back about a year ago, I ran into a problem. I had but a single IDE drive that was serving all of my Xen partitions. Now, like a dumbass, I overcommitted disk space on that IDE drive thinking "pah, I'll never use more than 20gigs on THAT machine!" How wrong I was. Needless to say, Xen was very unhappy and I don't think the IDE drive has ever recovered from being violated. It did something akin to writing over a second partition from the first partition. In hindsight, I should have filed a bug report with LKML. But that's another story. I came to the understanding that I couldn't half-ass my virtualization storage needs. I needed to get myself a SAN.

I searched the internet for a bunch of options and found that if I wanted anything remotely involving the letters S, A and N in any combination thereof from a commercial entity, I would have to sell my firstborn child. There must be another way!

Alas, the "another way" was suggested to me by my friend Frank -- I don't have much money to spend on this project, so why don't I use software raid with a bunch of cheapo disks? Project "alexandria" was born. I finally settled on getting a 4 drive hot swap cage from ebay, with 4 SATA 160gb drives. They were kinda expensive for my usual bargain-basement acquisitions, but I thought this would be a good idea and last me longer.

iSCSI happened almost by accident; I found that I had this awesome RAID5 storage array but no easy way to get the data to my Xen server. Sure I could use NFS to get the storage space exposed to the Xen machine, but then I'd still just be using files to store the filesystem. Frank, the everpresent techno-voice tells me I should really consider using LVM instead, since it allows some dynamic resizing and just lets you manage your space better. This was a great idea, but I still had no way to get those drives directly to Xen, short of doing NFS. I had heard of iSCSI previously from the technology world, and thought it'd be an excellent time to get my feet wet. I was surprised to find that iSCSI is drop dead simple to implement and use. I was expecting it to at least be difficult.

iSCSI at the very basic level has two pieces that intercommunicate: the system you're attempting to connect storage to (the Initiator) and the place where your storage resides (the Target.) For project Alexandria, I downloaded the iSCSI Enterprise Target software from sourceforge (http://iscsi-target.sourceforge.net.) The coolest thing about this software is the ease of compilation and use. As long as you have kernel headers, libc development libraries, and a compiler on the box, you're well on your way to having an iscsi target setup in about 3 minutes.

Once the software is compiled and installed, you'll want to fire up a target. The /etc/ietd.conf file that is supplied with the install is commented, and it's example target config is good for a first time user. (NOTE: many people advocate using CHAP to make people authenticate to the storage target! Personally I do not, but will shortly since my boxes are on ipv6 and I'd prefer not to have someone randomly login to my storage node and mess stuff up. You might also want to consider reading up on initiators.allow and initiators.deny in the iET documentation.)

Of note in the configuration is two major things you'll want to change:

1) The initiator name. (i.e., iqn.2001-04.com.example:storage.disk2.sys1.xyz) This is what your iscsi target will be named. The industry standard for naming is iqn.YYYY-MM.com.yourdomain: One thing to note is that the year and month are technically supposed to be the year and month you registered your domain (not sure why), but I'm sure you could get away with putting whatever the heck you wanted in there.

for example, one of my targets is named: iqn.2002-12.com.fierymoon:storage.w2k3.1

2) The Lun line. This line actually does the magic. For those who are not already educated in SCSI, a LUN is a Logical Unit Number. LUNs are used to differentiate different logical items inside of a single scsi device. For instance, certain CD changers that are scsi attached have multiple drives in them, so you'd have two cd-roms detected in SCSI on Bus Whatever, ID Whatever, LUN 0 and 1.

Anyway, the Lun line specifies what storage we're going to use. Since every scsi device that does not utilize LUN's are still technically considered to have only 1 logical unit, LUN 0, we have a line for Lun 0 in the ietd.conf:
Lun 0 Path=/dev/sdc,Type=fileio

What this line is saying is that the whole file of /dev/sdc (technically, the entire third SCSI disk in this linux host) should be available to any initiator that connects to the target. You can change this to be anything from a legitimate block device, to a LVM slice, or even a raw disk file. If you're following along on this and don't have a LVM or spare block device to use, do this:

dd if=/dev/zero of=/mynewlun bs=1M count=5000

This will create a new 5GB file at /mynewlun that you can then use as a "disk" for the purposes of iscsi. In this case, you'd change the LUN 0 line to read: "Lun 0 Path=/mynewlun,Type=fileio"

Once you've done that, save ietd.conf and fire up ietd: /etc/init.d/ietd start

Voila, hopefully you now have an iSCSI target. That actually was probably the hardest part of this exercise. Next post I'll talk about setting up an Initiator.