While pondering what my next project would be, I found myself evaluating what I have in my posession. I have two iSCSI SAN's, multiple machines running virtualization software. I've run up to 8 linux servers virtualized to run stuff in my HOUSE.
It's occured to me that most people aren't as balls-out crazy about computers as I am. This is quite a revelation.
Anyway, back to a very important thing that I should note in my blog. It is pretty difficult in the windows version of VMware's free Server product to utilize iSCSI devices as whole disks for VM's. In Linux, however, this is no problem at all. In the place where VMware asks you which physical hard drive to use, just put /dev/sda (or whatever iscsi device you have) and voila, you're utilizing iSCSI. There's really no way they can block you from this, as far as I know.
Also worth noting: DO NOT BUY THE PWSA8701T Intel iSCSI HBA! Well, let me qualify that. It works pretty darn well for $70. The problem is, they stopped supporting the driver for it in linux version 2.4. So, I'm not 100% pleased about it, but my vmware box is now running CentOS 2.1 to make use of the iSCSI HBA. I would have preferred some 2.6 love there, but concessions sometimes must be made.
Also in this week's errata: vmware does not like it when you hot-remove iscsi devices (i.e., rehashing ietd on your iscsi target box). Make sure you login to the vmware console and tell it to retry the disk operation it was attempting to do, or else your machine will "freeze" until someone answers the Retry. I wonder if there's a way to make it auto-retry. Normally in linux, if a drive has an I/O error, it will retry in (60?) seconds I believe. VMware obviously wants to be more nosy and let you know immediately there's something wrong.
Sunday, December 9, 2007
Monday, December 3, 2007
Random picture time!
Sunset in November makes some of the prettiest luminous cloud formations. Here's one of them I snapped from the parking lot at work.
Why you should never completely trust commercial products. (OR, never trust past employees of your company)
So, I brought home a scsi drive array from work that was sitting idle with 18G. It was at our colocation center and just sitting in a corner. My boss said I could have it. Yahoo! Free hardware.
That is, until I found "it didn't work." I've spent the last week or two tinkering, fiddling, figuring out if the problems with the drive array were with the card, the cables, the drives, or the case. If I had the drives in certain combinations, the array would partially work. If I had the drives in other slots, the whole thing would produce a loud shriek from my adaptec raid card.
Finally I was so annoyed by this problem that I popped open the case to the array and took a look inside. I was expecting to see resistors for drive ID's and thinking it must be a lost cause to tinker. Much to my happiness, I found that there were indeed jumpers for the scsi ID. What I wasn't prepared for was the configuration thereof.
For those who are not aware, scsi ID's are based on the binary code for 0 through 15. It works something like this:
0000 - SCSI ID 0
0001 - SCSI ID 1
0010 - SCSI ID 2
0011 - SCSI ID 3
And so on and so forth.
You'd expect that the manufacturer of the drive array would realize this to be the case. However, what greeted me was some very interesting math. It appeared that whoever set up the jumpers set them up to use 0,1,2,3 as decimal math as opposed to binary math. So, it was a creatively odd hodgepodge of scsi ID's in the case. I think they would have been able to pull it off, too, except for the fact that one of the scsi slots had real scsi ID 7, (0111, or 2+1+0).
That was the holy shitstorm my scsi card was raising. One of the freaking drive slots had scsi ID 7! For those not in the know, scsi ID 7 is traditionally the ID of the scsi card. When things collide on the SCSI bus with the adapter, weird things happen, (like the card spitting out that it had 32 IBM drives in it when it could only physically have 15).
Anyway, a simple bit of tweezers maintenance and the drive array is good-as-new, and is now rebuilding it's 8-disk RAID5 in preparation for me to migrate my Xen install to vmware.
Yahoo!
---------------------------
NOTE: Frank brought to my attention that it probably wasn't the factory, but the previous people who worked at my company who did this little feat of brilliance. Upon further investigation, the fact that the jumpers are of differing designs lends a great deal of credibility to that assumption. Therefore, Sorry <People Who Made Random Unlabeled Drive Array>, it really wasn't your people who were stupid.
That is, until I found "it didn't work." I've spent the last week or two tinkering, fiddling, figuring out if the problems with the drive array were with the card, the cables, the drives, or the case. If I had the drives in certain combinations, the array would partially work. If I had the drives in other slots, the whole thing would produce a loud shriek from my adaptec raid card.
Finally I was so annoyed by this problem that I popped open the case to the array and took a look inside. I was expecting to see resistors for drive ID's and thinking it must be a lost cause to tinker. Much to my happiness, I found that there were indeed jumpers for the scsi ID. What I wasn't prepared for was the configuration thereof.
For those who are not aware, scsi ID's are based on the binary code for 0 through 15. It works something like this:
0000 - SCSI ID 0
0001 - SCSI ID 1
0010 - SCSI ID 2
0011 - SCSI ID 3
And so on and so forth.
You'd expect that the manufacturer of the drive array would realize this to be the case. However, what greeted me was some very interesting math. It appeared that whoever set up the jumpers set them up to use 0,1,2,3 as decimal math as opposed to binary math. So, it was a creatively odd hodgepodge of scsi ID's in the case. I think they would have been able to pull it off, too, except for the fact that one of the scsi slots had real scsi ID 7, (0111, or 2+1+0).
That was the holy shitstorm my scsi card was raising. One of the freaking drive slots had scsi ID 7! For those not in the know, scsi ID 7 is traditionally the ID of the scsi card. When things collide on the SCSI bus with the adapter, weird things happen, (like the card spitting out that it had 32 IBM drives in it when it could only physically have 15).
Anyway, a simple bit of tweezers maintenance and the drive array is good-as-new, and is now rebuilding it's 8-disk RAID5 in preparation for me to migrate my Xen install to vmware.
Yahoo!
---------------------------
NOTE: Frank brought to my attention that it probably wasn't the factory, but the previous people who worked at my company who did this little feat of brilliance. Upon further investigation, the fact that the jumpers are of differing designs lends a great deal of credibility to that assumption. Therefore, Sorry <People Who Made Random Unlabeled Drive Array>
Thursday, November 29, 2007
iSCSI Initiator on Linux
I'm sorry I haven't posted this sooner, it's a situation of "when something's this easy, does it even require writing down?" However, I promised it and so I will deliver. Using Linux as an iSCSI Initiator isn't all that much harder than windows. It's just a matter of getting the right command line arguments.
You want to either get a precompiled package of the open-iscsi project, or download and compile it yourself. The open-iscsi software is pretty easy to compile as well, just do the standard make && make install and you're rockin-and-rollin.
The primary tool for talking to the open-iscsi kernel modules is iscsiadm. iscsiadm is a bit difficult to learn at first so I'll give some quick info.
Q. How do I discover targets?
iscsiadm -m discovery -t sendtargets -p <target's ip>
Q. How do I login to a target I've discovered?
iscsiadm -m node -T <target's name> -p <target's ip> --login
This will create a new scsi drive on your system, check dmesg to see where it put it. I actually use udev on elysium to make it easier to find iscsi mounts by chain,id,lun rather than /dev/sdX.
The same command with --logout will remove the drive from your system.
Once it's on your system, you're ready to treat it like any other scsi disk. fdisk, mkfs.ext3 and rock and roll!
You want to either get a precompiled package of the open-iscsi project, or download and compile it yourself. The open-iscsi software is pretty easy to compile as well, just do the standard make && make install and you're rockin-and-rollin.
The primary tool for talking to the open-iscsi kernel modules is iscsiadm. iscsiadm is a bit difficult to learn at first so I'll give some quick info.
Q. How do I discover targets?
iscsiadm -m discovery -t sendtargets -p <target's ip>
Q. How do I login to a target I've discovered?
iscsiadm -m node -T <target's name> -p <target's ip> --login
This will create a new scsi drive on your system, check dmesg to see where it put it. I actually use udev on elysium to make it easier to find iscsi mounts by chain,id,lun rather than /dev/sdX.
The same command with --logout will remove the drive from your system.
Once it's on your system, you're ready to treat it like any other scsi disk. fdisk, mkfs.ext3 and rock and roll!
ISN/Freenum and why you should care.
I wanted to take a moment and discuss a cool telephony technology that I think everyone should be aware of. Why? It's going to change the world.
From the freenum.org website:
"The Freenum/ISN system is 12-digit-keypad (telephone handset) friendly method of providing mapping between users. While the eventual use of email-style URI pointers is the eventual goal for communications identifiers, it is still the case that the majority of the world's telephony users are trapped using a 12-digit keypad for extended numeric entry, and it is for the purposes of these devices that the Freenum system and ISN dialing was designed as a "stopgap" which may last many years. Using ENUM-like methods and IETF standards but not using telephone numbers, the Freenum/ISN dialing system is designed to initially allow SIP-capable proxies and iPBX systems to connect to each other in a free, open, and protocol agnostic manner over the Internet. Initially focused on SIP voice communications, the platform is hoped to eventually extend to other communications methods as those protocols become more widespread."
What does this mean in plain, simple English? It means that using this technology any person in the world can communicate to any other person in the world for no more than the cost of the bandwidth of the call and a little tech savvy. There are plenty of free VoIP clients (X-Lite, SJPhone, Zoiper just to name a few.) If you have asterisk or know someone who does (or any sip proxy, for that matter) you can likely piggyback on their system for ISN resolution.
Let me explain a bit about how this works such that a layperson might understand. Lets say I have relatives in, I dunno, Cambodia (Or insert random country here.) They have high speed internet access and so do I. How do I call them cheaply? Sure, I could use something like vonage and pay per-minute rates. But why bother? We can use the internet from one side of the call to the other, with no issues.
Now, a lot of people might be saying "But you can just use the remote computer's IP address and put that in the client to connect!" Yes, of course you can. The thing that makes ISN pretty cool, however, is that instead of remembering an ip address or a hostname, you only have to remember a relatively small number that is similar to a phone number. In my case, 230*739 will get my desk at home. In the previous case, you would have to type a line similar to sip:ip1-5-7-2-blah.core1.cambodia.has.long.dns.entries.kh or something. Even if you used their dynamic dns service (that you had to teach them how to use) it'd still be sip:petescambodianfamily.dyndns.org. Wouldn't you rather dial something like 1234*800?
The easiest method to use isn is to get on a free SIP provider such as Free World Dialup. They already have ISN built into their sip proxies and all you had to do was download a sip client and sign up for the free service. Now, granted, you could have both you and your cambodian buddies get free world dialup and never even have to use ISN. But, then you'd be missing out on learning about a new technology in the process that might just make your life easier!
ISN is designed to utilize DNS for it's resolution. When you dial 1234*800, what's really happening on the proxy you are connected to is, they're doing what's called an ENUM lookup to 4.3.2.1.800.freenum.org. This lookup produces (via a NAPTR dns entry) a URI that can be used to call. For example:
pgrace@fmservices:~$ host -t NAPTR 0.3.2.739.freenum.org
0.3.2.739.freenum.org NAPTR 100 10 "u" "E2U+sip" "!^\\+*([^\\*]*)!sip:\\peter.grace@hermes.fierymoon.com!" .
This record in simplified English just tells the sip proxy that it needs to call sip:peter.grace@hermes.fierymoon.com whenever someone dials 230*739 via ISN. The best part is, most people will never need to know how this works. It will "Just Work", thanks to you, the techie of the household.
There are plenty of people on the internet and in the isn community that will be happy to help you setup ISN if you already have a working sip proxy. There are also many people who will help you setup an asterisk box if you have a bit of linux skills and are willing to learn the nuances of voip.
It gets better though.
Lets say you have kids in college. You want to be able to call them from home cheaply, and they want to hit you up for cash regularly. Who wants to pay money for these calls? It's like you're really paying them 102.50$ for every 100$ you put in their bank account.
Luckily for you, a lot of universities have already signed up for ISN. They will be rolling out their extensions via ISN to link into their campus phone networks soon if they have not already done so. What that means is, your child who has an extension in their dorm room of say, 2325, all you'll have to do is dial 2325*<itad>. Lets say your kid goes to Wayne State University in Detroit (Random college I pulled from the ITAD List). Their ITAD is 533. By dialing 2325*533, voila, you have just connected to your irate daughter as she was about to go clubbing with her roomies, and the best part is, you're not paying for the call!
Little Jenny needs some more money? If you signed up for Free World Dialup or any of the other myriad sip providers that support ISN, all she'd need to do is dial <your fwd number>*262 and voila, her nice little neon lightup phone from the 80's has made a voip call to your PC.
There is a caveat to this, and that is that ISN is still considered early-adopters territory. Not everyone listed in the ITAD list has a legitimate termination setup for ISN. Some places have flakey sip proxies. This is definitely more bleeding-edge digital convergence stuff than some people are used to dealing with.
That being said, my asterisk box is always up, go on ahead and give me a call at 230*739!
From the freenum.org website:
"The Freenum/ISN system is 12-digit-keypad (telephone handset) friendly method of providing mapping between users. While the eventual use of email-style URI pointers is the eventual goal for communications identifiers, it is still the case that the majority of the world's telephony users are trapped using a 12-digit keypad for extended numeric entry, and it is for the purposes of these devices that the Freenum system and ISN dialing was designed as a "stopgap" which may last many years. Using ENUM-like methods and IETF standards but not using telephone numbers, the Freenum/ISN dialing system is designed to initially allow SIP-capable proxies and iPBX systems to connect to each other in a free, open, and protocol agnostic manner over the Internet. Initially focused on SIP voice communications, the platform is hoped to eventually extend to other communications methods as those protocols become more widespread."
What does this mean in plain, simple English? It means that using this technology any person in the world can communicate to any other person in the world for no more than the cost of the bandwidth of the call and a little tech savvy. There are plenty of free VoIP clients (X-Lite, SJPhone, Zoiper just to name a few.) If you have asterisk or know someone who does (or any sip proxy, for that matter) you can likely piggyback on their system for ISN resolution.
Let me explain a bit about how this works such that a layperson might understand. Lets say I have relatives in, I dunno, Cambodia (Or insert random country here.) They have high speed internet access and so do I. How do I call them cheaply? Sure, I could use something like vonage and pay per-minute rates. But why bother? We can use the internet from one side of the call to the other, with no issues.
Now, a lot of people might be saying "But you can just use the remote computer's IP address and put that in the client to connect!" Yes, of course you can. The thing that makes ISN pretty cool, however, is that instead of remembering an ip address or a hostname, you only have to remember a relatively small number that is similar to a phone number. In my case, 230*739 will get my desk at home. In the previous case, you would have to type a line similar to sip:ip1-5-7-2-blah.core1.cambodia.has.long.dns.entries.kh or something. Even if you used their dynamic dns service (that you had to teach them how to use) it'd still be sip:petescambodianfamily.dyndns.org. Wouldn't you rather dial something like 1234*800?
The easiest method to use isn is to get on a free SIP provider such as Free World Dialup. They already have ISN built into their sip proxies and all you had to do was download a sip client and sign up for the free service. Now, granted, you could have both you and your cambodian buddies get free world dialup and never even have to use ISN. But, then you'd be missing out on learning about a new technology in the process that might just make your life easier!
ISN is designed to utilize DNS for it's resolution. When you dial 1234*800, what's really happening on the proxy you are connected to is, they're doing what's called an ENUM lookup to 4.3.2.1.800.freenum.org. This lookup produces (via a NAPTR dns entry) a URI that can be used to call. For example:
pgrace@fmservices:~$ host -t NAPTR 0.3.2.739.freenum.org
0.3.2.739.freenum.org NAPTR 100 10 "u" "E2U+sip" "!^\\+*([^\\*]*)!sip:\\peter.grace@hermes.fierymoon.com!" .
This record in simplified English just tells the sip proxy that it needs to call sip:peter.grace@hermes.fierymoon.com whenever someone dials 230*739 via ISN. The best part is, most people will never need to know how this works. It will "Just Work", thanks to you, the techie of the household.
There are plenty of people on the internet and in the isn community that will be happy to help you setup ISN if you already have a working sip proxy. There are also many people who will help you setup an asterisk box if you have a bit of linux skills and are willing to learn the nuances of voip.
It gets better though.
Lets say you have kids in college. You want to be able to call them from home cheaply, and they want to hit you up for cash regularly. Who wants to pay money for these calls? It's like you're really paying them 102.50$ for every 100$ you put in their bank account.
Luckily for you, a lot of universities have already signed up for ISN. They will be rolling out their extensions via ISN to link into their campus phone networks soon if they have not already done so. What that means is, your child who has an extension in their dorm room of say, 2325, all you'll have to do is dial 2325*
Little Jenny needs some more money? If you signed up for Free World Dialup or any of the other myriad sip providers that support ISN, all she'd need to do is dial <your fwd number>
There is a caveat to this, and that is that ISN is still considered early-adopters territory. Not everyone listed in the ITAD list has a legitimate termination setup for ISN. Some places have flakey sip proxies. This is definitely more bleeding-edge digital convergence stuff than some people are used to dealing with.
That being said, my asterisk box is always up, go on ahead and give me a call at 230*739!
Tuesday, November 27, 2007
From LAN Storage With Love
So now we have our cheap "SAN." Yes, at this point you can call it a SAN because you are using SAN technology to talk to it. I don't want to hear people complain that "it's not redundant!" You know what, I can pull both plugs on your expensive qlogic array and you'll be just as screwed.
Anyway, I digress. Since we've got an iSCSI Target setup, we'll need to do something with it. If you are running Windows, google for the "Microsoft iSCSI Initiator." It will be the first hit on the page, I can almost guarantee it. Download the software and install. It's a JKHN (Just Keep Hitting Next) situation, with a reboot at the end (probable, not definite.) Once your machine reboots, double-click the icon on the desktop for the Microsoft iSCSI Initiator. As an aside, this icon is just a shortcut the control panel applet, so don't get your panties in a bunch that it's going to stop working if you close out the program. It merely talks to the iscsi initiator stack that was installed in windows with the program.
The first step is to click the Discovery tab on the screen that pops up. From there, click Add. Put the IP address of your iscsi-enabled machine in the ip box and hit ok. Since we didn't tell ietd to run on a different port, it is using the default assigned port for iscsi.
Depending on the orientation of the sun and other factors, it may take a sec for the box to go away. It's connecting to the IP and getting a target list. Click the Target tab. If for some reason there aren't targets displayed, hit Refresh. Once you do that, your new target you defined on the linux box should now proudly display! Click that entry in the target list and hit Log On. It'll bring up a box about multipathing IO and whether or not you want to have the drive persist at boot. I would leave Multipathing IO alone for another, more advanced discussion on iSCSI failover technology. Suffice to say you don't need it for your little proof-of-concept.
OK, so you've clicked the OK button. Nothings happening! No magic bells and whistles will go off, nor fireworks to show that you've instantly got a new disk. This is because all windows knows is that a new, EMPTY scsi drive was just connected.
From here, we have to go to Logical Disk Manager Service in computer management, from administration tools, from control panel. (OR, if you're lazy like me, "diskmgmt.msc" in your Run box.) The disk management wizard will automatically fire up telling you it's seen a new disk in your machine that needs to be converted/initialized. You will want to initialize the disk, but not convert it. Once you've done this, the panel will tell you that there's a disk available that is Unallocated. Right click the disk and hit properties and make sure it's listed as an "IET VIRTUAL DISK" -- We want to make sure we're not wasting your active drive, now do we?
From here, you can partition the drive and assign it a drive letter. voila, you're now using iscsi in windows, and all it took us was like, 20 minutes max (if you did this while reading along.)
Once you get accustomed to it, you can have new drives available in mere seconds! Especially when I tell you how to use iSCSI Enterprise Target to dynamically add drives without restarting the daemon. After all, we wouldn't want to have to shut down all our systems just to add a new target! That'll be a post in another day or so. Next up is how to do this same procedure with a linux machine.
Anyway, I digress. Since we've got an iSCSI Target setup, we'll need to do something with it. If you are running Windows, google for the "Microsoft iSCSI Initiator." It will be the first hit on the page, I can almost guarantee it. Download the software and install. It's a JKHN (Just Keep Hitting Next) situation, with a reboot at the end (probable, not definite.) Once your machine reboots, double-click the icon on the desktop for the Microsoft iSCSI Initiator. As an aside, this icon is just a shortcut the control panel applet, so don't get your panties in a bunch that it's going to stop working if you close out the program. It merely talks to the iscsi initiator stack that was installed in windows with the program.
The first step is to click the Discovery tab on the screen that pops up. From there, click Add. Put the IP address of your iscsi-enabled machine in the ip box and hit ok. Since we didn't tell ietd to run on a different port, it is using the default assigned port for iscsi.
Depending on the orientation of the sun and other factors, it may take a sec for the box to go away. It's connecting to the IP and getting a target list. Click the Target tab. If for some reason there aren't targets displayed, hit Refresh. Once you do that, your new target you defined on the linux box should now proudly display! Click that entry in the target list and hit Log On. It'll bring up a box about multipathing IO and whether or not you want to have the drive persist at boot. I would leave Multipathing IO alone for another, more advanced discussion on iSCSI failover technology. Suffice to say you don't need it for your little proof-of-concept.
OK, so you've clicked the OK button. Nothings happening! No magic bells and whistles will go off, nor fireworks to show that you've instantly got a new disk. This is because all windows knows is that a new, EMPTY scsi drive was just connected.
From here, we have to go to Logical Disk Manager Service in computer management, from administration tools, from control panel. (OR, if you're lazy like me, "diskmgmt.msc" in your Run box.) The disk management wizard will automatically fire up telling you it's seen a new disk in your machine that needs to be converted/initialized. You will want to initialize the disk, but not convert it. Once you've done this, the panel will tell you that there's a disk available that is Unallocated. Right click the disk and hit properties and make sure it's listed as an "IET VIRTUAL DISK" -- We want to make sure we're not wasting your active drive, now do we?
From here, you can partition the drive and assign it a drive letter. voila, you're now using iscsi in windows, and all it took us was like, 20 minutes max (if you did this while reading along.)
Once you get accustomed to it, you can have new drives available in mere seconds! Especially when I tell you how to use iSCSI Enterprise Target to dynamically add drives without restarting the daemon. After all, we wouldn't want to have to shut down all our systems just to add a new target! That'll be a post in another day or so. Next up is how to do this same procedure with a linux machine.
My beginning steps in iSCSI
Back about a year ago, I ran into a problem. I had but a single IDE drive that was serving all of my Xen partitions. Now, like a dumbass, I overcommitted disk space on that IDE drive thinking "pah, I'll never use more than 20gigs on THAT machine!" How wrong I was. Needless to say, Xen was very unhappy and I don't think the IDE drive has ever recovered from being violated. It did something akin to writing over a second partition from the first partition. In hindsight, I should have filed a bug report with LKML. But that's another story. I came to the understanding that I couldn't half-ass my virtualization storage needs. I needed to get myself a SAN.
I searched the internet for a bunch of options and found that if I wanted anything remotely involving the letters S, A and N in any combination thereof from a commercial entity, I would have to sell my firstborn child. There must be another way!
Alas, the "another way" was suggested to me by my friend Frank -- I don't have much money to spend on this project, so why don't I use software raid with a bunch of cheapo disks? Project "alexandria" was born. I finally settled on getting a 4 drive hot swap cage from ebay, with 4 SATA 160gb drives. They were kinda expensive for my usual bargain-basement acquisitions, but I thought this would be a good idea and last me longer.
iSCSI happened almost by accident; I found that I had this awesome RAID5 storage array but no easy way to get the data to my Xen server. Sure I could use NFS to get the storage space exposed to the Xen machine, but then I'd still just be using files to store the filesystem. Frank, the everpresent techno-voice tells me I should really consider using LVM instead, since it allows some dynamic resizing and just lets you manage your space better. This was a great idea, but I still had no way to get those drives directly to Xen, short of doing NFS. I had heard of iSCSI previously from the technology world, and thought it'd be an excellent time to get my feet wet. I was surprised to find that iSCSI is drop dead simple to implement and use. I was expecting it to at least be difficult.
iSCSI at the very basic level has two pieces that intercommunicate: the system you're attempting to connect storage to (the Initiator) and the place where your storage resides (the Target.) For project Alexandria, I downloaded the iSCSI Enterprise Target software from sourceforge (http://iscsi-target.sourceforge.net.) The coolest thing about this software is the ease of compilation and use. As long as you have kernel headers, libc development libraries, and a compiler on the box, you're well on your way to having an iscsi target setup in about 3 minutes.
Once the software is compiled and installed, you'll want to fire up a target. The /etc/ietd.conf file that is supplied with the install is commented, and it's example target config is good for a first time user. (NOTE: many people advocate using CHAP to make people authenticate to the storage target! Personally I do not, but will shortly since my boxes are on ipv6 and I'd prefer not to have someone randomly login to my storage node and mess stuff up. You might also want to consider reading up on initiators.allow and initiators.deny in the iET documentation.)
Of note in the configuration is two major things you'll want to change:
1) The initiator name. (i.e., iqn.2001-04.com.example:storage.disk2.sys1.xyz) This is what your iscsi target will be named. The industry standard for naming is iqn.YYYY-MM.com.yourdomain: One thing to note is that the year and month are technically supposed to be the year and month you registered your domain (not sure why), but I'm sure you could get away with putting whatever the heck you wanted in there.
for example, one of my targets is named: iqn.2002-12.com.fierymoon:storage.w2k3.1
2) The Lun line. This line actually does the magic. For those who are not already educated in SCSI, a LUN is a Logical Unit Number. LUNs are used to differentiate different logical items inside of a single scsi device. For instance, certain CD changers that are scsi attached have multiple drives in them, so you'd have two cd-roms detected in SCSI on Bus Whatever, ID Whatever, LUN 0 and 1.
Anyway, the Lun line specifies what storage we're going to use. Since every scsi device that does not utilize LUN's are still technically considered to have only 1 logical unit, LUN 0, we have a line for Lun 0 in the ietd.conf:
Lun 0 Path=/dev/sdc,Type=fileio
What this line is saying is that the whole file of /dev/sdc (technically, the entire third SCSI disk in this linux host) should be available to any initiator that connects to the target. You can change this to be anything from a legitimate block device, to a LVM slice, or even a raw disk file. If you're following along on this and don't have a LVM or spare block device to use, do this:
dd if=/dev/zero of=/mynewlun bs=1M count=5000
This will create a new 5GB file at /mynewlun that you can then use as a "disk" for the purposes of iscsi. In this case, you'd change the LUN 0 line to read: "Lun 0 Path=/mynewlun,Type=fileio"
Once you've done that, save ietd.conf and fire up ietd: /etc/init.d/ietd start
Voila, hopefully you now have an iSCSI target. That actually was probably the hardest part of this exercise. Next post I'll talk about setting up an Initiator.
I searched the internet for a bunch of options and found that if I wanted anything remotely involving the letters S, A and N in any combination thereof from a commercial entity, I would have to sell my firstborn child. There must be another way!
Alas, the "another way" was suggested to me by my friend Frank -- I don't have much money to spend on this project, so why don't I use software raid with a bunch of cheapo disks? Project "alexandria" was born. I finally settled on getting a 4 drive hot swap cage from ebay, with 4 SATA 160gb drives. They were kinda expensive for my usual bargain-basement acquisitions, but I thought this would be a good idea and last me longer.
iSCSI happened almost by accident; I found that I had this awesome RAID5 storage array but no easy way to get the data to my Xen server. Sure I could use NFS to get the storage space exposed to the Xen machine, but then I'd still just be using files to store the filesystem. Frank, the everpresent techno-voice tells me I should really consider using LVM instead, since it allows some dynamic resizing and just lets you manage your space better. This was a great idea, but I still had no way to get those drives directly to Xen, short of doing NFS. I had heard of iSCSI previously from the technology world, and thought it'd be an excellent time to get my feet wet. I was surprised to find that iSCSI is drop dead simple to implement and use. I was expecting it to at least be difficult.
iSCSI at the very basic level has two pieces that intercommunicate: the system you're attempting to connect storage to (the Initiator) and the place where your storage resides (the Target.) For project Alexandria, I downloaded the iSCSI Enterprise Target software from sourceforge (http://iscsi-target.sourceforge.net.) The coolest thing about this software is the ease of compilation and use. As long as you have kernel headers, libc development libraries, and a compiler on the box, you're well on your way to having an iscsi target setup in about 3 minutes.
Once the software is compiled and installed, you'll want to fire up a target. The /etc/ietd.conf file that is supplied with the install is commented, and it's example target config is good for a first time user. (NOTE: many people advocate using CHAP to make people authenticate to the storage target! Personally I do not, but will shortly since my boxes are on ipv6 and I'd prefer not to have someone randomly login to my storage node and mess stuff up. You might also want to consider reading up on initiators.allow and initiators.deny in the iET documentation.)
Of note in the configuration is two major things you'll want to change:
1) The initiator name. (i.e., iqn.2001-04.com.example:storage.disk2.sys1.xyz) This is what your iscsi target will be named. The industry standard for naming is iqn.YYYY-MM.com.yourdomain:
for example, one of my targets is named: iqn.2002-12.com.fierymoon:storage.w2k3.1
2) The Lun line. This line actually does the magic. For those who are not already educated in SCSI, a LUN is a Logical Unit Number. LUNs are used to differentiate different logical items inside of a single scsi device. For instance, certain CD changers that are scsi attached have multiple drives in them, so you'd have two cd-roms detected in SCSI on Bus Whatever, ID Whatever, LUN 0 and 1.
Anyway, the Lun line specifies what storage we're going to use. Since every scsi device that does not utilize LUN's are still technically considered to have only 1 logical unit, LUN 0, we have a line for Lun 0 in the ietd.conf:
Lun 0 Path=/dev/sdc,Type=fileio
What this line is saying is that the whole file of /dev/sdc (technically, the entire third SCSI disk in this linux host) should be available to any initiator that connects to the target. You can change this to be anything from a legitimate block device, to a LVM slice, or even a raw disk file. If you're following along on this and don't have a LVM or spare block device to use, do this:
dd if=/dev/zero of=/mynewlun bs=1M count=5000
This will create a new 5GB file at /mynewlun that you can then use as a "disk" for the purposes of iscsi. In this case, you'd change the LUN 0 line to read: "Lun 0 Path=/mynewlun,Type=fileio"
Once you've done that, save ietd.conf and fire up ietd: /etc/init.d/ietd start
Voila, hopefully you now have an iSCSI target. That actually was probably the hardest part of this exercise. Next post I'll talk about setting up an Initiator.
Monday, November 26, 2007
But first, some information about myself.
I thought that before I get into some of the nitty-gritty of the miscellaneous topics I'll cover, that I should probably explain in more detail who I am and why I'm worth reading. Simply put, I'm not really worth reading unless you're a fan of stream of consciousness and hardware/software hackery. Some of my interests include virtualization, storage technologies, amateur radio, VoIP, and playing Guitar Hero. I am very proficient in Linux (I've been using it for 13 years now) and am also pretty proficient in using technology in ways other than how they were originally intended.
I'm married, have two cats and a house, and am trying to live the American Dream, even if sometimes it seems like it's just a fleeting daydream rather than a good "Let-Me-Fall-Back-Asleep" kind of dream.
I'm married, have two cats and a house, and am trying to live the American Dream, even if sometimes it seems like it's just a fleeting daydream rather than a good "Let-Me-Fall-Back-Asleep" kind of dream.
How To Spend A Lot Of Money On Your Electric Bill
It's important for me to introduce the cast of characters regarding my future blogs of random stuff. Let me begin with a setting:
"It was a dark, stormy night in Pete's office. Dark, Stormy nights are particularly bad for Pete's Office, since he has a lot of technology but no battery backup. Dark, Stormy nights are also difficult for Pete because if the power goes out, he trips over all the crap in his office getting out."
The cast:
elysium - 8U Dell PowerEdge 6330 (It weighs a freaking ton!) - Xen system
alexandria - 4U mini-SAN (iSCSI, LVM, Athlon with 384mb of ram)
vmhost01 - big tower (soon to be 4U) dual-proc athlon MP box with 2gb ram - vmware server.
virgil - athlon x64 desktop.
fmusptwsan001 -- New san box, big scsi cards with cylon-eye-like LED's.
4U Drive array with 8x18GB 10k scsi drives connected to fmusptwsan001
hamshack - Old p3 for my ham radio stuff.
hermes - IBM Netfinity 1U dual-proc p3 for asterisk, dhcp, some other misc goodies.
Now, you may ask yourself: "Why on earth does he have all this equipment but no battery backup?" The answer? I'm cheap. At some point I'll invest in a rackmount 3kVa ups for my little setup, but until then I live dangerously. Hell, it's just my local LAN.
...Which is part of the reason why I'm writing this blog. Old surplus hardware can be made to do amazingly interesting things. That's what I'm all about.
"It was a dark, stormy night in Pete's office. Dark, Stormy nights are particularly bad for Pete's Office, since he has a lot of technology but no battery backup. Dark, Stormy nights are also difficult for Pete because if the power goes out, he trips over all the crap in his office getting out."
The cast:
elysium - 8U Dell PowerEdge 6330 (It weighs a freaking ton!) - Xen system
alexandria - 4U mini-SAN (iSCSI, LVM, Athlon with 384mb of ram)
vmhost01 - big tower (soon to be 4U) dual-proc athlon MP box with 2gb ram - vmware server.
virgil - athlon x64 desktop.
fmusptwsan001 -- New san box, big scsi cards with cylon-eye-like LED's.
4U Drive array with 8x18GB 10k scsi drives connected to fmusptwsan001
hamshack - Old p3 for my ham radio stuff.
hermes - IBM Netfinity 1U dual-proc p3 for asterisk, dhcp, some other misc goodies.
Now, you may ask yourself: "Why on earth does he have all this equipment but no battery backup?" The answer? I'm cheap. At some point I'll invest in a rackmount 3kVa ups for my little setup, but until then I live dangerously. Hell, it's just my local LAN.
...Which is part of the reason why I'm writing this blog. Old surplus hardware can be made to do amazingly interesting things. That's what I'm all about.
Metablogging
Yes, I am starting a blog. No, I have no idea exactly what I'm going to put on it. Deek of 555 fame says I should blog about the plethora of random/weird projects I find myself working on in my off time. I think that's a good suggestion, I'm going to start documenting some of the random stuff I get my kicks with.
Subscribe to:
Posts (Atom)