Starwind/iSCSI and small block sizes

[Thoro]

Hi,

I've two completly identical servers both with 2 x 300 GB SSDs the following are the local transferrates with ATTO Disk Benchmark:

256MB Total Length and QD: 4

0.5: 22912 22854
1.0: 43008 45965
2.0: 79908 79933
4.0: 138990 142054
8.0: 222188 241302
16.0: 227721 289383
32.0: 222564 324967
64.0: 226178 317587
128.0: 221890 316560
256.0: 215646 310671
512.0: 225576 440058
1024.0: 225576 451694
2048.0: 220029 490293
4096.0: 227970 499414
8192.0: 224163 493674

If I connect both Servers to the HA and test again with ATTO i get:

0.5: 1221 3137
1.0: 2295 6144
2.0: 4729 12800
4.0: 9924 25848
8.0: 18244 47104
16.0: 30913 84423
32.0: 61392 148376
64.0: 91840 249853
128.0: 93128 321900
256.0: 174457 436540
512.0: 277883 646832
1024.0: 540474 693143
2048.0: 718440 787585
4096.0: 704723 901222
8192.0: 486957 1094341

As you see the values from 0.5 to 256 aren't really good compared to the lokal disks - is there anyway to improve the performance??

I've enabled MPIO with 5 connections to each server, Jumbo Frames, everything that has to do with Offloading.

They are connected over 2 x 10 GBit/s Ethernet and also get full speed with netio (2 instances ~ 1100 MB/s per port)

Servers are:
Dual E5640 (2.4 GHz HexaCore)
48 GB RAM
Server 2008 R2 Enterprise SP1 and all updates

thanks for any help!!

best regards
Thomas Rosenstein

[anton (staff)]

Picture you've provided is incomplete:

1) What target type do you use? CDP? D/D? IMG? Do you have write-back cache enabled? Of what size?

2) What OS and what initiator do you use on your client? Do you have properly working MPIO?

P.S. It's better to attach screenshots as well.

[Thoro]

1) I use: Hard Disk / High Availability - write back cache is enabled with 1024 MB, and 5000 ms cache expiry period

MPIO is activated and all connections say "Active/optimized"

there are 5 to each server - the MPIO Policy is set to Round Robin

2) I'm using Windows Server 2008 R2 Enterprise and the default Microsoft iSCSI Initiator

I'll compose a few screenshots for you and send you the downloadlink per PM

edit: Somehow I can't send you PMs ?!

so here it is:

you can download the screenshots from here: http://www.thoro.at/starwind.zip

I've included a few ATTO Test with different settings

1 local (127.0.0.1) iSCSI Connection
1 remote iSCSI Connection
5 local iSCSI Connections
5 remote iSCSI Connections
5 local and 5 remote iSCSI Connections
local hard disk

then screenshots from Starwind and the iSCSI Dialog and also the MPIO Dialog

[anton (staff)]

This is strange... Do you use all-around-10 GbE connections for all of the uplinks?

No idea why you cannot send PM but you'd better attach images here - forum allows this. Let us check the images before..

[anton (staff)]

Now could you please comment attached screenshots. What's where and what test did you run. Thank you!

[Thoro]

Okay,

the images are all from Server 1

Starwind_local.PNG

Screenshot of the "storage.img" configuration

starwind_local.PNG

Second Screenshot of the "storage.img" config

iSCSI.PNG

Image of the iSCSI Window where 5 connections are made locally and 5 remote to the second server

iSCSI_Properties.PNG

Detail of one of the iSCSI Connections (all look the same, except that for the local server there are 5 MPIO Connections and for the remote server there are the 5 sync connections + 5 MPIO Connections)

ATTO_local_disk.PNG

A standard ATTO run on the local disk to get reference values

ATTO_iSCSI_1local.PNG

A standard ATTO run with only 1 iSCSI Connection to the locally running Starwind Server

ATTO iSCSI_1remote.PNG

A standard ATTO run with only 1 iSCSI Connection to the remote running Starwind Server (second server)

ATTO_iSCSI_5local.PNG

A standard ATTO run with 5 iSCSI Connections to the local Starwind Server with MPIO enabled

ATTO_iSCSI_5remote.PNG

A standard ATTO run with 5 iSCSI Connections to the remote running Starwind Server (second server) with MPIO enabled

ATTI_iSCSI_5local_5remote.PNG

A standard ATTO run with 5 iSCSI connections to the local Starwind Server and 5 iSCSI Connections to the remote Starwind Server (second server) with MPIO enabled

-------------------------

The Servers are directly connected over a 2 Port Intel X520-T Network Adapter, cable length is 0,5 m Cat 6A, with NetIO I get around 1120 MB/s per Port - I haven't bonded them

[Thoro]

Here are the other 3 test runs

I've no problem to provide you remote access additional images etc. I just need to know what you'd like to have, because I don't really know what exactly to look for

[anton (staff)]

Please let us check everything and then we'll probably go remote session way.

[lodiver]

Hey Thoro

Did you try this tweek:
http://sustainablesharepoint.wordpress. ... ith-iscsi/

starwind-f5/looking-for-some-peer-support-performance-problem-t2386.html

hope it helps

[Thoro]

Hi Lodiver,

i've applied the change but it didn't change ANYTHING

to check it again, I've tested with a RAMDRIVE of 5 GB here are my results:

the Access Time is quite good with 0.5 ms, but the small reads/writes just plain suck

[anton (staff)]

Compare local I/O and "networked" I/O. Separate numbers tell nothing.

[Thoro]

yes that's why I made the first tests with the local hard disk test and the networked test to see what the difference is - I've read somewhere that the difference should be about ~10% - is that correct?


then I made the RAM Disk test to completly ignore the physical disk hardware - but also that drive is slow with the small data chunks, so either the network adapter is doing something wrong or the iscsi initiator or starwind

I don't really have comparison values with other systems, but I think my results are odd



I do have another question: is there any way to improve the loopback performance?

[anton (staff)]

No it's not correct. You do insert a latency for network and request processing. So on fully booked pipeline when you fire multiple requests in a deep queue there will be no difference in local and networked I/O. But for a short "bursts" you need to add network latency and request processing latency on both sides. Can result 2x - 4x performance degradation.

yes that's why I made the first tests with the local hard disk test and the networked test to see what the difference is - I've read somewhere that the difference should be about ~10% - is that correct?


then I made the RAM Disk test to completly ignore the physical disk hardware - but also that drive is slow with the small data chunks, so either the network adapter is doing something wrong or the iscsi initiator or starwind

I don't really have comparison values with other systems, but I think my results are odd



I do have another question: is there any way to improve the loopback performance?

[Thoro]

Okay so after reading a little bit it seems ATTO is not really an appropriate benchmark (only a queue of 4)

So I've tried SQLIO with two threads and a queue of 8 per thread

Sizes of:

1k
4k
64k

Command Line: (once for size for local and once for each size for remote)
sqlio.exe -kR -s60 -fsequential -o8 -b1 -LS -FC:\param timeout /T 10
sqlio.exe -kR -s60 -frandom -o8 -b1 -LS -FC:\param timeout /T 10
sqlio.exe -kR -s60 -fsequential -o8 -b1 -LS -FC:\param timeout /T 10
sqlio.exe -kR -s60 -frandom -o8 -b1 -LS -FC:\param timeout /T 10

I've used a simple Image File, and two iSCSI Connection over the two X520T Ports with enabled MPIO

64k full speed for random and sequential
4k half speed at sequential, full speed at random
1k quarter at sequential, full speed at random

so anton, do you think there is a way to also get full speed for the 1k and 4k sequentials test? Or is this due to overhead?

And through the various tests I've noticed one thing that really bugs me:

MS Initiator Access Times

HDTUNE_Access_Times_6_remote_dual_con_nocache.PNG (46.08 KiB) Viewed 425 times

Starwind StarPort Access Times

Starwind_Initator_HDTUNE_Access_Times_1_remote.PNG (48.55 KiB) Viewed 423 times

As you see the MS Initiator takes quite a time on the max. Access Times, with your (the Starwind StarPort Initiator) the maximum access times are much better, but not as good as on the real device (add network lat. etc)

I've attached the outputfiles as a zip archive

[anton (staff)]

There's no way to get wire speed with small requests.