Smaller Jumbo Frames?

[Aitor_Ibarra]

In my brief performance tests of latest beta, I don't see any difference between 4088 byte and 9016 byte jumbo frames. With either, I was able to get to 550MB/sec reads from cache over 10GbE, and one test when I had two clients testing the same target simultaneously got to 700+ MB/sec.

So, if your clients are using the default windows allocation unit of 4096 bytes (which is more than a 4088 byte frame!) and you use 9k frames, and on the starwind side your disk is also formatted as 4096 byte allocation units, is the ethernet 9016 byte packet padded with zeroes, or will it have two allocation units worth of data in it? Or is the allocation unit size irrelevant because iSCSI groups IOs into larger chunks, or whatever?

Basically I'm looking for a reason for there not being much of a difference between 4088 byte and 9016 byte frames, and wondering what the optimal setting is really going to be given that most IOs are going to be for multiples of 4096 bytes. In the end I actually care more about IOPs than transfer rates, but it's that 4096 byte windows default that I want to optimise for.

[anton (staff)]

Jumbo frame size is entirely implementation-specific thing and depends on the selected configuration greatly. The same hardware but different drives can change everything (Intel Pro 1GbE hardware for example, 16KB Vs. 9KB performance chances both sides from release to release). Also intermediate network hardware (switches) have great influence. So proper answer is: TEST BEFORE YOU PUT INTO PRODUCTION

In my brief performance tests of latest beta, I don't see any difference between 4088 byte and 9016 byte jumbo frames. With either, I was able to get to 550MB/sec reads from cache over 10GbE, and one test when I had two clients testing the same target simultaneously got to 700+ MB/sec.

So, if your clients are using the default windows allocation unit of 4096 bytes (which is more than a 4088 byte frame!) and you use 9k frames, and on the starwind side your disk is also formatted as 4096 byte allocation units, is the ethernet 9016 byte packet padded with zeroes, or will it have two allocation units worth of data in it? Or is the allocation unit size irrelevant because iSCSI groups IOs into larger chunks, or whatever?

Basically I'm looking for a reason for there not being much of a difference between 4088 byte and 9016 byte frames, and wondering what the optimal setting is really going to be given that most IOs are going to be for multiples of 4096 bytes. In the end I actually care more about IOPs than transfer rates, but it's that 4096 byte windows default that I want to optimise for.

[deiruch]

Awakening zombie thread!

Overhead of a 9k frame compared to a 4k frame is only a little lower. And if you enable jumbo frames not all frames are padded to a certain size - the max size is just bigger. So 20 byte packets won't suddendly be 9k after enabling jumbo frames. It's just that 9k packets don't have to be fragmented & reassembled from multiple 1.5k parts.

Having 4k jumbo frames means 2 packets per 4k read (Because of iScsi overhead and data), if you enable 9k frames a single packet should be enough. Grouping cannot be done with iSCSI.

[anton (staff)]

Sure. MTU is *Maximum* Transfer Unit and not constant.

Again... The answer is "it depends". So both 4KB and 9KB packets should be checked before putting whole configuration into production.

Awakening zombie thread!

Overhead of a 9k frame compared to a 4k frame is only a little lower. And if you enable jumbo frames not all frames are padded to a certain size - the max size is just bigger. So 20 byte packets won't suddendly be 9k after enabling jumbo frames. It's just that 9k packets don't have to be fragmented & reassembled from multiple 1.5k parts.

Having 4k jumbo frames means 2 packets per 4k read (Because of iScsi overhead and data), if you enable 9k frames a single packet should be enough. Grouping cannot be done with iSCSI.