Starwind HA Cluster + HyperV

[Max (staff)]

great! *deleted* I'll upgrade both nodes now

One by one! otherwise both HA servers will be out of sync,
Install 5.6 on the second - wait for sync to finish - update first.
However if it's just a test environment - remove everything -> update everything-> recreate everything

[hixont]

Any idea when the Most Recent Version thread will be updated with the version 5.6 release news? I only know about newer release because this thread is of interest to me.

[anton (staff)]

We did not have *official* V5.6 release yet. Scheduled for next week.

Any idea when the Most Recent Version thread will be updated with the version 5.6 release news? I only know about newer release because this thread is of interest to me.

[starneo]

I upgraded both starwind storages now and dird some testing.
The good news is, I had no BSOD / Starwind service crash till now.
The bad news is, I do now have performance issues.

For testing the performance of the cluster I run Sisoft Sandra 2011 in a VM, before the update to 5.6 I had ~470MByte Read (random and sequence) and ~240MByte write (random and sequence). Now I got ~470MByte read and 35MByte write. It is not only the benchmark showing this issue, during simpel filecopy I see the same, usually less then 10MByte.

For me it looks like an issue with the sync channel.

I will do some more testing, but I need some help on this.
I did not change something else but to update to 5.6.

[starneo]

I tested a full sync with one target, its very small because it contains the hypervquorum. Its 10GB.
I attached a screenshot of my syncperformance, as you see it is juming up and down in average 2-3% of 10Gbit.
Normaly it has to be 10% of 10Gbit, and it was ... sometimes.

*edit*

There is something else I recognized. In version 5.5 the RAM on both starwind nodes droped when I booted one. In 5.6 it keeps at max load.
And in 5.5 the RAM load increased when using the targets one by one, in 5.6 the RAM load goes to its max during the starwind startup.
I think this is intend to be so?

[starneo]

I now do a full sync of all 6 targets you can see below how slow it is. It should be at ~60% ...
I now have no more ideas ...
I need some help with this.

[starneo]

Okay guys, I found the solution for the sync problem.

I had to enable the Receive Side Scaling (RSS) in the Myricom 10GBe Network Card.
The RSS queue is 4.
The global settings posted here http://www.starwindsoftware.com/forums/starwind-f5/recommended-tcp-settings-t2293.html did not work (I have set this settings directly after the first starwind installation!).
The settings were not set for the Myricom Network Adapter.

If you wish I can post all my network card settings for the sync adapter.

[@ziz (staff)]

For sure, it would be great if you make the post!
Thanks!

[starneo]

To be clear, this resolved the sync speed issue, as far as I can tell for now.
If my performance issue is resolved I can test tomorrow - the full sync will taks ~8h.
Something else I noticed, I started the HyperV Cluster and 9 VMs on it. The full sync is still in progress.
Ths sync speed seems to be affected ny this, it drops a little bit during the VM start (to arround 40%) but is now at 50% again.

Okay here are my settings, these settings might be specific for myricom:

flow control = on
Interrupt coalescing delay = 25
IPv4 checksum offload = Rx enabled
Large send offload v2 (ipv4) = enabled
Large send offload v2 (ipv6) = enabled
log link state event = enabled
MTU = 9000
receive buffers = 2048
Receive side scaling (rss) = enabled
rss queues = 4
strip vlan tags = on
TCP & UDP checksum offload (ipv4) = rx & tx enabled
TCP & UDP checksum offload (ipv6) = rx & tx enabled

[@ziz (staff)]

Thank you for the information and please keep us updated about further tests results.

[mkaishar]

Should flow control be enabled?
In reading numerous publications they state that flow control could slow down network performance.
So I usually disable all flow control on the SAN networks...curious if that is relevant?

[Max (staff)]

Flow Control disabling does not always give normal results with iSCSI. In my experience this was one of 6-7 infrastructures with nearly identic hardware.

[mkaishar]

Flow Control disabling does not always give normal results with iSCSI. In my experience this was one of 6-7 infrastructures with nearly identic hardware.

I am sorry I don't understand what you mean here.
Enabling it is a good idea with iSCSI?
Disabling it is a good idea with iSCSI?

Thanks,
Mark

[Max (staff)]

I meant that there is no 100% guarantee that flow control switching will help you. This option needs to be tested individually.