Write issues on v8 (currently 7774)

[Trevbot]

ESXi 5.5 VM SSD write speed to Thick Disk (imagefile) with no cache (L1 or L2) setup on v8-7774 (also occurred on 7509) running on Server 2012 is extremely poor. 50MBps best case. Reads are 400-500+MBps and I have tried on 4 different SSD types (V300, M500, 840EVO, 840PRO). I deleted the Thick Disks(s) ran secure erase on the SSDs and re-created the Thick Disks but the write performance is still slow. I deleted all Thick Disks and uninstalled v8. I installed an old copy of v6-6.0.5713 and created new Thick Disks, again with no caching. I was surprised to see that both read and write performance were back on par with what I would expect (250-400MBps for both read and write depending on the SSD) so I am not sure where to go from here. v6 runs at the speeds I expect but does not have the VAAI support I want. v8 has the VAAI support but at the cost of write performance. I uninstalled v6 and reinstalled v8-7774. Recreated the Thick Disks and, not surprisingly, still slow writes. I am open to any troubleshooting and or configuration suggestions anyone may have. Thanks!

TL;DR
ESXi 5.5 Host.
Server 2012 R2 VM.
10Gb link to SW SAN.
SW SAN running on Server 2012 on all SSD (no raid).
Poor Write speeds on v8 but great on v6.
Read speeds great on both v8 and v6.
All ESXi iSCSI settings and SW SAN v8 (and v6) settings at defaults.

[Max (staff)]

Hi Trevbot,
Thank you for your input, we really appreciate it.
That's definitely not the behavior we expect to see in an all-SSD environment.
Is there any chance our engineers could get remote access to the systems to dob the performance tuning and benchmarking?
Feel free to PM me or drop an e-mail referencing this forum thread to support@starwind.com

[Trevbot]

Hi Max,

Sorry for the delay, I sent you an email today in regard to setting something up to have an engineer look at the system(s). Quick Update from some tests ran over the weekend:

original v8-7774 box:
Server 2012
FLAT devices good reads, slow writes
LSFS devices good reads and writes
LSFS defrag observed working after storage vMotion off LSFS to FLAT device
all devices on SSD storage (2 FLAT 2 LSFS) no raid, connected via LSI 9211-8i

new test v8-7774 box:
Server 2012
FLAT and LSFS devices good reads and writes
both devices on SSD storage (1 FLAT 1 LSFS) no raid, directly connected to SATA ports

tests:
deployed VMs to/on the v8-7774 boxes with MDT and SCCM 2012R2 and observed bandwidth and IOPS during the deployment @ the SW Mgmt Console and Performance Monitor for disk activity on and wire traffic to the San.
ran ATTO and HDTune benchmarking for various length (ATTO incremental up to 1GB and HDTune 500MB to 5000MB with a mix of zero, mixed and random) on deployed VMs

The 2 machines have diff processors and original has 16GB RAM and new has 32GB but neither is registering high utilization during reads or writes. Original currently has V300 and M500 SSD and the new has Agility4 (not known for their blazing speed but performing as I would expect). I can swap SSDs between systems and the LSI controller is variable I can test to see if directly plugging the SSD improves speed but it's odd that LSFS reads and writes fine with the controller and in v6 reads and writes were good also.

[Max (staff)]

Hi Trevbot!
Glad to hear the new tests showed better performance.
As for the variable, did you use any cache for the SSDs on the LSI level?
Our R&D has some ideas regarding the performance differences between v6 and v8 you got.
We'll be able to verify them in ~ 3 days.

[Trevbot]

Hi Max,

no L1 or L2 cache in v8 or v6 and the LSI controller has no onboard cache.

[Anatoly (staff)]

Any chance you could share with us benchmarking specifications and output for both tests please?

[Trevbot]

Hi Anatoly,

I had run some tests with ATTO and HDTune and emailed the results to Max. Yesterday we got on a remote session and ran another battery of tests using ATTO and IOMeter. Max was going to check with the engineers in regard to the write performance we saw. I can loop you in on any additional communication and if you like, I can email you the screen snips I sent him.

[Max (staff)]

Quick community update.
We're currently investigating the issue with FLAT image performance issue.
Suspecting alignment issues causing performance drops on small block sizes.
Either me or OP will provide an update as soon as we get more details.

[Max (staff)]

Community update:
Removing the LSI adapter from the equation and connecting the disks directly to the MB results in expected performance results.
We currently suspect internal alignment issues in the LSI controller causing these issues.