Recommendations for a Large SAN for Virtualization

[bdford]

I am looking to build a large HA storage pool (anywhere from 10-40TB) which will be used as primary storage for around 200 XEN server instances. The XEN instances are for use by our clients and as such, there functions will vary. We foresee that around 80% of the instances will be Windows based and could act as Terminal Servers, small Exchange and SQL servers, application, and file servers. Of-course, some clients will build Linux instances for website hosting, MySQL databases, etc.

Since we really have no control over what the instances will be used for, I am trying to spec out an array that will walk the line between performance and scalability, all while not breaking budget. The server I am planning on using will contain 3 LSI MegaRAID SAS 9260-4i controllers, 36 600GB SAS 15K drives, with the option of adding capacity using an additional 36 drive JBOD. The server itself is a XEON E5504 with 6GB of memory. This setup will be replicated for the second server for HA purposes.

We are currently toying with the idea of using RAID50, which gives us much more bang for the buck when it comes to storage capacity, but I'm a little worried about the performance hit that we will suffer when compared to RAID10. Can anyone comment on whether or not this config looks adequate to meet our vague requirements? Any comments in regards to RAID10 vs RAID50 performance?

Thank in advance for any advise.

[Aitor_Ibarra]

Hi,

I use Starwind as a SAN for a hyper-v cluster (I rent virtual machines), and although I don't have as many VMs as you, and am using a different hypervisor, have the same sort of issue as you in terms of needing to accomodate lots of VMs. I'm also in the middle of switching from Areca 1680ix-24 to LSI 9280 RAID cards...

OK.. with 3 raid cards in one server, you will have potentially killer i/o. BUT, 9260-4i card is probably the wrong card for you. 4i will directly connect to 4 drives, and you want to do 36 drives, which is 12 drives per card. You can do this, with a SAS expander - maybe your server chassis has one built in, make sure it is LSI based for compatibility, but in a way you are wasting the bandwidth of the SAS 2108 chip that card uses, because it can power 8 drives at full bandwidth. The main advantage you gain with the three cards is 3 x the cache, but this is less useful if you are going to use Starwind's write back cache. Also, even with SAS expanders, the 9260 series is firmware limited to 32 drives max. So, if you expect to run large numbers of drives, I recommend the 9280 series, which are the same, but have a higher drive limit (240 per card), and at least one external x4 port for JBODs. Anyway, a single 9260 or 9280 is easily capable of saturating one or two 10G iSCSI connections - given enough fast drives.

Next, choice of drive - I kind of think that 15K SAS drives are in a bad value segment now. The cost per GB is high enough for you to look at SSDs, where cost per IOPs is much better than 15K SAS. So, consider instead using 10K SAS drives and a high performance tier of SSD. Put your high IOPs LUNS on SSD, middle IOPs on 10K SAS, and maybe get a low IOPs tier of 7.2K SAS.

If you go LSI and get some SSDs, you may want to look at the Fast Path and CacheCade firmware options. These cost extra, but will get more performance from SSDs, either in SSD only LUNs (FastPath), or using the SSDs as a read cache for hot data (CacheCade) - you can use a RAID 0 of up to 32 SSDs / 512GB to cache an HD based RAID volume.

RAID levels: if you've got lots of VMs sharing the same spindles, I think you are best off with either big RAID 10s (if you can put the virtual hard drives of multiple VMS on to one LUN), or multiple RAID-1s (if you want to give each virtual hard drive a dedicated LUN/spindles). The disadvantage is that with LSI you cannot do online capacity expansion of RAID-10 (but same goes for 50 and 60), so you can't just add disks to grow capacity - you're going to have to shuffle some data around. RAID 5 and 6 do support online capacity expansion, although performance will suffer as the RAID needs to be rebuilt. Having said that, with these cards, the bottleneck will be the drives, not the card, so if you can take the SAN offline during a rebuild, the rebuild time will be defined by the speed of one drive, and not by parity calculations. RAID-10's performance during a rebuild tends to be better as only two drives are involved, so less thrashing.

For me, RAID level boils down to paranoia vs convenience. Paranoia says - I want minimum number of customers / vms to be impacted by a drive failure - so multiple RAID-1. Convenience says a single big LUN is much easier to manage - so RAID-10.

I wouldn't touch 5 or 50 because of the increased risk of the entire RAID failing if 2nd drive fails during / before a rebuild. The more,larger drives you have, the greater the risk. Also, I think in your environment you will care more about IOPs than sequential transfer speeds, so advantage of 60 vs 6 is minimal. So I think your choice is between 1,10 and 6.

Oh - and then double your SAN hardware budget as unless you can do planned downtime for stuff like windows updates, you *really* need HA.

Hope this helps!

cheers,

Aitor

[bdford]

Aitor -

Thanks for the lengthy explanation. This is exactly the kind of stuff I am looking for. I am going to read over everything a bit more and may have some follow-up questions, but I wanted to go ahead and clarify that I will be running two of these SANS for HA - so my budget is already doubled.

Originally we priced out 10K drives instead of the 15K. However, the price difference wasn't enough to justify not making the jump. I will look into the SSD options a bit more. I honestly didn't even consider SSD as I assumed the price jump would be too much to stomach.

Off to do some digging. Thanks again for the recommendations. I look forward to other opinions as well.

[Aitor_Ibarra]

If you are looking at 600GB @ 15K then that probably means 3.5" drives, which I haven't costed - maybe the premium for them isn't so high as with 2.5". I tend to use 3.5" just for 2TB "big, slow and cheap" tier and 2.5" for everything else.

Also the choice of SSDs that are actually suitable for this is quite small. If you go for MLC instead of SLC based SSD for cost reasons, you really need to overprovision to maintain performance and longevity. Even with 50% kept empty, MLC is cheaper than SLC. And there are very few SSDs with SAS interfaces... which is a shame as SAS is in my opinion a much better interface for SSD than SATA. The current gen Sandforce based SAS SSDs just do SATA <-> SAS so there's no real advantage. So if you want SAS, that leaves you with Pliant (cost per GB is very high!!) and soon Hitachi.

If you can hold off your purchase till Q1 2011 you will have a lot more choice and better prices in SSD - P300 from Micron (SLC version of Crucial C300, but still SATA), new Intel SSDs (up to 600GB MLC, cheaper per GB), and Hitachi (SAS SLC).

[awedio]

Aitor,

Curious why you are switching to the 9280 & not the Areca 1880x?

...I'm also in the middle of switching from Areca 1680ix-24 to LSI 9280 RAID cards..

[anton (staff)]

I'm not Aitor but I personally try to stay away from Areca cards. They have too many issues reported to support. Far more then any other vendor. And issues are non-StarWind related. Like for example random 1-2 second (yes, SECOND) I/O delays under heavy load. Enough to spoil test results...

[rchisholm]

I know 1680's with the BIOS versions before 1.47 had problems with I/O under load, but have you had problems with them on 1.47 and the max payload size for the PCI-E slots set to 128? I did some very extensive testing with IOPS reaching over 100k for extended periods of time on these cards using 12 Intel X25-E SSD's on each of them and never ran into any strange I/O delays. If you are running into all these problems with at least version 1.47, I guess I'll have to consider myself very lucky. Worries me a little since I have 4 of the 24 port 1880's sitting here new in boxes getting ready to be installed. Of course, they will never be under extreme I/O loads since they will only end up with 24 7.2K SAS drives each.

[anton (staff)]

We had issues ~2 years ago so if firware had been fixed it's fine. But negative impression is still present

I know 1680's with the BIOS versions before 1.47 had problems with I/O under load, but have you had problems with them on 1.47 and the max payload size for the PCI-E slots set to 128? I did some very extensive testing with IOPS reaching over 100k for extended periods of time on these cards using 12 Intel X25-E SSD's on each of them and never ran into any strange I/O delays. If you are running into all these problems with at least version 1.47, I guess I'll have to consider myself very lucky. Worries me a little since I have 4 of the 24 port 1880's sitting here new in boxes getting ready to be installed. Of course, they will never be under extreme I/O loads since they will only end up with 24 7.2K SAS drives each.

[rchisholm]

Been there, done that. Having worked as a systems engineer that did contract work for over 100 companies for 10 years, there are definitely some hardware manufactures I won't even consider for a second. Don't care how much they improved, I don't want anything to do with them.

We had issues ~2 years ago so if firware had been fixed it's fine. But negative impression is still present

I know 1680's with the BIOS versions before 1.47 had problems with I/O under load, but have you had problems with them on 1.47 and the max payload size for the PCI-E slots set to 128? I did some very extensive testing with IOPS reaching over 100k for extended periods of time on these cards using 12 Intel X25-E SSD's on each of them and never ran into any strange I/O delays. If you are running into all these problems with at least version 1.47, I guess I'll have to consider myself very lucky. Worries me a little since I have 4 of the 24 port 1880's sitting here new in boxes getting ready to be installed. Of course, they will never be under extreme I/O loads since they will only end up with 24 7.2K SAS drives each.

[Aitor_Ibarra]

Aitor,

Curious why you are switching to the 9280 & not the Areca 1880x?

On one of my two Starwind servers, both of which had 1680 cards, I ran into a situation where the server would not boot because the Areca wouldn't recognise the cards attached to it. Some of the time. Actually, most of the time, and initially it seemed that the card wouldn't even allow the system to POST until I removed it from the slot and reinserted it. I tested the card on a different server and it was fine, so sent back card and server to supplier, who tried lots of things, like changing power suppliers and motherboard, but couldn't make the problem go away.

Now given the fact that I had to upgrade, and also the fact that Areca's support wern't particularly helpful, I decided to switch to LSI. I'd not used an LSI RAID card before, but had plenty of of experience with their motherboard RAID and also LSI clone cards that Dell use (the PERC cards have been LSI for years since they switched from Adapatec). Finally, the LSI SAS-2 cards had been out for a year, with very positive reviews, whereas Areca's SAS-2 range had suffered major delays (they were going to use a Marvell ROC, which was ARM based like the Intel ROC used on the 1680) but were forced to switch late in the day to the LSI 2108, which powers the LSI cards too..

I've not seen the high i/o bugs with Areca, and other than this problem, it's always been a solid performer. However, I've never used RAID5 or 6 with it, and there is a problem with the lack of threading in the firmware which is supposed to affect performance with multiple RAID5/6 volumes. I had multiple RAID 1/10 and never saw this.

LSI are much easier to obtain in the UK, and seem to have better support. I miss a few features from Areca, but LSI compensates with better features elsewhere. I also like that the BBU can be mounted to the card.

The main drawbacks - well, LSI aren't yet compatible with an HP SAS expander which is very popular as it provides loads of SAS ports for a low price. Areca is compatible. Also, Areca have an out-of-band management port, which is very useful, although I don't have issues with LSI's in-band management (in fact, I prefer it, but this is a matter of opinion). And I miss 4GB cache, but with Starwind now using system RAM as cache, this is less of an advantage.

[awedio]

Aitor,

I thought you used SuperMicro chassis? Did you switch to something else.
The SMicro expanders are LSI compatible

The main drawbacks - well, LSI aren't yet compatible with an HP SAS expander which is very popular as it provides loads of SAS ports for a low price. Areca is compatible. Also, Areca have an out-of-band management port, which is very useful, although I don't have issues with LSI's in-band management (in fact, I prefer it, but this is a matter of opinion). And I miss 4GB cache, but with Starwind now using system RAM as cache, this is less of an advantage.

[Aitor_Ibarra]

I thought you used SuperMicro chassis? Did you switch to something else.
The SMicro expanders are LSI compatible

No, still with Supermicro chassis and motherboards, and yes, all Supermicro SAS stuff these days is LSI based. One of the reasons I switched to LSI rather than stayed with Areca is that Supermicro support LSI RAID cards with their JBODs etc but not Areca.

Areca 1880 might be different to 1680 with regards to LSI compatibility, given that it runs on an LSI ROC, but I did have issues with the 1680 and a Supermicro JBOD (SAS-1) which have gone away by switchng to LSI.

[oxyi]

I don't feel that great reading this post now...

I just finished building 2 50x 2tb Chenbro chassis server, each server is powered by 2 x Areca 1680 4Gb cache version, with 1.48 firmware.

How do I go about testing the high I/O load to see if my areca would have a problem, we still in testing stage so I can check for it...

Thanks !

[anton (staff)]

Please keep us updated if possible. Your feedback on the topic is very much appreciated! Thank you!

I don't feel that great reading this post now...

I just finished building 2 50x 2tb Chenbro chassis server, each server is powered by 2 x Areca 1680 4Gb cache version, with 1.48 firmware.

How do I go about testing the high I/O load to see if my areca would have a problem, we still in testing stage so I can check for it...

Thanks !

[oxyi]

Sure, but I need to know if there is tool that I can use to stress test the IO and what's the setting for it.