Building an Inexpensive iSCSI SAN on Centos 6

I was recently given an HP DL360 G4 server with two 36GB SCSI disks. At first blush the system is of little use in todays tech landscape due to it’s low storage capacity and limited processor feature set. My unit has two Intel Xeon processors at 3.0Ghz without Intel’s VT technology and 4GB of RAM. It would make a good single use server for many functions and would grow in usefulness as it’s storage capacity increased. SCSI hardware has fallen out of general use in recent years and has been superseded by SAS, or Serial Attached SCSI, in much the same way that for desktop machines SATA has replaced PATA. We’ve gotten to a point where moving one stream of data really fast is more efficient than moving several streams together but at lower speed.

With this in mind, I started doing a little research on Ebay for larger capacity SCSI drives that I could swap into this machine. I was able to find mountains of SCSI disks for pretty reasonable prices but again and again the HP Storage Works 30, better known as the HP MSA30, kept popping up. There are loads of them on Ebay these days fully loaded with 36GB, 73GB, or 146GB SCSI disks. I did a little research and discovered that the MSA30 is a 14 disk SCSI enclosure with no real onboard intelligence; It’s just a box with 14 hot-swap SCSI disks and a SCSI port or two on the back. The unit supports the final SCSI protocol that saw any real success in the market, SCSI Ultra-320, meaning it can move data at 320MB/s. The MSA30 had the option of being configured with a single SCSI bus for all 14 disks or two buses, each handling 7 drives. Depending on your needs you would pick the configuration that would suit you best. It is possible to fully saturate a SCSI bus so if you were concerned about the bus being a bottleneck, you could improve you theoretical throughput by splitting the disks across two buses and having 640MB/s of throughput available. My intention is to build a SAN that operates over a Gigabit network though, which means the theoretical maximum speed for me is 120MB/s. Bus saturation is not a concern.

With the MSA30 in mind I began to do some research on the supported SCSI RAID cards available. The DL360 G4 server has two full height PCI-X slots, one is full length and the other is half length. If you’ve never seen a full length card before, they are more than 12″ long and not the size you’re likely to see in a desktop computer. HP was selling the Smart Array 6400 series cards with the MSA30 units. The cards with external ports, the 6402 and 6404, are the ones I was interested in. The 6402 has two external SCSI VHDCI ports and the 6404 has four, both are full length cards to the DL360 G4 can only house one of these cards.

With my research complete I took to Ebay and started buying up parts. When all was said and done, two MSA30s were purchased plus a Smart Array 6402. I also planned to use an Adaptec 29320 card I had lying around. It’s a PCI-X card but it’s not full length and will comfortable fit into the server with the 6402. After a quick stop over at Monoprice.com for 3′ SCSI VHDCI cables to connect the MSA30’s to the SCSI cards, it was time to begin the build.

I began by setting up the DL360 with Centos 6, my current favorite linux distribution. I did a minimal installation and added in the necessary bits to install and setup the HP tools I needed for management. I wanted to be able to use the web based RAID configuration tool as well as the other web based system management tools they make available. It wasn’t too hard to do but it’s likely worth it’s own post. From there I started to provision all of this new storage.

One of the MSA30s I purchased was loaded with 14 146GB 10k disks and a single SCSI bus. Since this was to be the lower performance unit I hooked it up to the Adaptec card and disabled all of it’s onboard RAID tools. The unit supports RAID1, RAID1 and RAID10 but it wasn’t able to utilize all 14 disks in a single RAID10, which was my goal, so I used the linux based MDADM software RAID tools instead. I created on large RAID10 which totals just shy of 1TB of storage. Performance testing tells me that I am able to write to the enclosure at, on average, 130MB/s and I can read at 222MB/s. It’s likely that I could do some performance tuning to increase that a bit but since it’s faster than the 1Gb link the iSCSI SAN will run it, there is no reason to.

The second of the MSA30s came with 14 146GB 15k disks and two SCSI buses. This was to be the high performance unit so I wired up each 7 disk group to a port on the Smart Array 6402 and I used the HP RAID config tools to build the RAID10 array. The card I bought has a working battery backup so I enabled write caching, which in concert with the faster disks and dual SCSI buses provided higher performance than the first array. When tested identically to the first array the second MSA30 give an average write speed of 169MB/s. It’s read speed though is twice as fast as the other array and comes in at just about 460MB/s. Had I been using a single SCSI bus that could have been no faster then 320MB/s so there is certainly a performance improvement for the dual-bus enclosure. Depending on the work load you plan to put on the MSA30 getting the dual bus enclosure could certainly make sense.

From there it was a simple matter to configure the NICs on the server and then setup the iSCSI server. I have one NIC on my main network and the second is on a separate network dedicated to just iSCSI traffic. I used this guide (http://www.server-world.info/en/note?os=CentOS_6&p=iscsi) to setup the iSCSI server though I didn’t use logical volumes.

The HP DL360 G4 server I used was given to me so it cost me nothing but they sell on Ebay these days for $40-$60 dollars. I bought two MSA30 units and I paid $140 a piece for them with free shipping. I paid $18 for the Smart Array 6402 and just shy of $70 for the three SCSI VHDCI cables I needed to wire everything up. In total, the whole system cost me less than $400, which is incredibly cheap for a high performance iSCSI SAN, even if I do say so myself!