HA with 10gbit intel nics CX4

[Jerky_san]

I know this has been posted many times but I can't seem to find anything about actually tuning a nic. Basically I've tried reading all the posts about slow HA performance before posting. I went and got the latest intel driver for this card in both servers(v16). I taken the switch completely out of the loop by directly hooking the 2 servers together with a single CX cable. I've applied all settings recommended in the sticky for tcpip.

Boxes are identical

8 core opteron
8 gigs of ram
10 15k rpm 450 gig Hitachi sas - Raid 6(will probably change) - Read Rate - 700 Megabytes Per Second - Write Rate - 730 Megabytes per Second.
Areca raid 24 port sas raid cards
2 7200 RE3 WD 500 gig for OS
Windows 2008 R2 Standard

I've been using tttcp to measure network between the 2 boxes.. By doing this I've found that they will only talk to each other at a 6500Mbits and averages 5500Mbits. That is an attempt at totally saturation by creating many buffers / connections. If I do a single connection the max is about 2000mbits and averages about 1800mbits. If I do a test on the hard drive from the opposite server I get 125 megabyte read/110megabyte write. If I do it to an HA disk its 56.6 megabytes read and 11.9 megabytes write. Through reading the forums I've seen performance should be half which it is at least on the read side. Currently though my concern is actually with the servers talking to one another fast. I figure if I fix that issue it will fix the rest of the issues. So basically I was wondering if any of you on this forum have tuning knowledge for intel 10gbit CX4 nics? I've tried everything I could think of like enabling jumbo packets(no gains), uping RSS Queues to 16, ensure that the IRQ's are not sharing at all, and doing all the tweaks I've read on this forum. Any help is appreciated.

[anton (staff)]

Do you have 10 GbE "all-around" or just between HA nodes? Could you please provide us with your infrastructure interconnection diagram? For now the numbers you've provided are OK for 1 GbE and for 10 GbE they just suck (( Let's find out why ))

[Jerky_san]

Yes, when we decided to go with starwind software's HA setup it was also decided that we would do 10gb nics to all servers that will connect to the HA SAN system and to each other. The VM system we are currently running is hyper-v and both of them have 10gb nics to each other and to the SAN. The HA SAN system is in the same boat. 10gb nics to each other and to the VM system. My network diagram looks like what is in the "Hyper-V and High Availability Shared Storage" white paper on page 7. I have been attempting to follow all the docs to a T. Down to even having two switches(we will have one around Monday or Tuesday. The modules just aren't in but the body is. The second is planned in the near future when budgets allow). The only thing that is different currently is that the two cluster machines are directly linked and so is the two SAN's instead of letting the information pass through a switch. Since I do not have the switch yet I'm currently testing communication between the clusters and between the HA. The CX4 cables are HP since the switch itself is an HP 5406 pro curve.

[anton (staff)]

We have everything pretty much the same as you do and (surprise) it works as required. Is it possible to get a Remote Desktop session to your config? If you don't keep WikiLeaks mirror of course... Could save us all tons of time if we could jump directly in. Thanks!

Yes, when we decided to go with starwind software's HA setup it was also decided that we would do 10gb nics to all servers that will connect to the HA SAN system and to each other. The VM system we are currently running is hyper-v and both of them have 10gb nics to each other and to the SAN. The HA SAN system is in the same boat. 10gb nics to each other and to the VM system. My network diagram looks like what is in the "Hyper-V and High Availability Shared Storage" white paper on page 7. I have been attempting to follow all the docs to a T. Down to even having two switches(we will have one around Monday or Tuesday. The modules just aren't in but the body is. The second is planned in the near future when budgets allow). The only thing that is different currently is that the two cluster machines are directly linked and so is the two SAN's instead of letting the information pass through a switch. Since I do not have the switch yet I'm currently testing communication between the clusters and between the HA. The CX4 cables are HP since the switch itself is an HP 5406 pro curve.

[camealy]

I assume this went offline to get fixed, if it was could the remedy be posted?

Thanks,

Kurt

[Max (staff)]

Hi, I have been Working with Jerky San on this issue. The low performance error is residing on the Hyper-V side
An iSCSI device connected to the client gives about 80% of the array speed, the fix for this is pending in the 5.7 by implementing MCS workaround and spawning more workers to work via 10 GbE networks.
The actual performance drop is obsrerved on the VM which is working on the target device. The issue is obviously caused by partition allignment and is now submitted both to our Q&A dept. and Microsoft support.
We are expecting a solution within a week.
I want to highlight the importance of block allignment once more: 95% of best practice configs have the RAID stripe size = Parent NTFS block size = child NTFS block size = 64k. Child NTFS block size of 512b also gives better results than default (4k)
The NIC settings for CX4 have been reset to factory in this particular case.

[camealy]

So are you saying that the raid array should be 64K stip size, the parent NTFS partition (direct attached to the Starwind host) should be 64K NTFS, and then the VM partition inside the vhd that will be inside the .img should be 512b?

Thanks,

Kurt

[Max (staff)]

So are you saying that the raid array should be 64K stip size, the parent NTFS partition (direct attached to the Starwind host) should be 64K NTFS, and then the VM partition inside the vhd that will be inside the .img should be 512b?

I said that the client side NTFS block size has to be either 64k or 512b - file transfer and benchmarking with iometer shows best results there. The low performance issue is observed in the VMs which reside on this volume. A 2008r2 VM on the iSCSI disk gives 3-5Mb/s write performance but the underlying iSCSI disk itself is working on the top speed (limited either by NIC or array throughput)

[camealy]

So does the NTFS partition cluster size on the Starwind host that the .img's sit on matter?

Thanks,

Kurt

[Max (staff)]

I think we've got a misunderstanding here:
64K stripe size on the SAN raid->64k block size of the NTFS SAN volume->Default Image file exported as iSCSI disk->64k/512b Client side server block size = good performance
As soon as you create a Hyper-V VHD on top of this - writes drop to 2-3 MB/s
So it's not IMG but VHD problem