Enable 9K Jumbo Frames for all your NICs both on the target and initiator side. Jumbo Frames should also be enabled for all switches.
Note: Sometimes, Jumbo Frames may degrade performance and should be disabled in such cases.
How to enable Jumbo Frames:
Microsoft Windows 2012, 2012 R2, 2016
Control Panel -> System -> Device Manager -> Your NIC -> Advanced.
Microsoft Windows 2012, 2012 Server Core, 2016, 2016 Server Core, Hyper-V Server
To enable Jumbo Frames on Server Core editions of Windows, run the following commands:
netsh int show int
This command will list names of NICs. Afterward, run the following command for each NIC:
netsh int ipv4 set subint "<Name of NIC>" mtu=9000 store=persistent
VMware ESX(i)
To set up Jumbo Frames for vSwitch, run the following command in a console:
esxcfg-vswitch -m 9000 <vSwitch>
To create a VMKernel with Jumbo Frames support, run the following commands:
Note: If you change Jumbo Frames on the existing vSwitch, it does not affect the packet size of the VMKernel. To change the packet size of the VMKernel, delete the existing VMKernel and create a new one.
esxcfg-vswitch -d
esxcfg-vswitch -A vmkernel# vSwitch#
esxcfg-vmknic -a -i <ip address> -n <netmask> -m 9000 <portgroup name>
(The detailed information can be found here: VMWare KB article 1007654))
Citrix XenServer
To enable Jumbo Frames, download vSwitch Controller Virtual Appliance from My Citrix Page, import it to the XenServer and run it. After performing the above-mentioned actions, navigate to the XenServer host console and type the following:
xe-switch-network-backend openvswitch
xe vif-param-set uuid=<vif_uuid> other-config:mtu=9000
To get <vif_uuid>, use xe vif-list command.
(More detailed information is provided in XenServer 5.6 Feature Pack 1 Administrator's Guide and XenServer 5.6 Feature Pack 1 vSwitch Controller User Guide)
Windows tweaks:
Microsoft Windows 2012, 2012 Server Core, 2016, 2016 Server Core, Hyper-V Server
These commands should be run from the command prompt with admin rights.
netsh int tcp set heuristics disabled - disables Windows scaling heuristics
netsh int tcp set global autotuninglevel=normal - turns-on TCP auto-tuning
netsh int tcp set global congestionprovider=ctcp - turns on using Compound TCP
netsh int tcp set global ecncapability=enabled - enables ECN
netsh int tcp set global rss=enabled - enables Receive-side Scaling (RSS). Note, that you should enable it only if your NIC supports it!
netsh int tcp set global chimney=enabled - enables TCP Chimney offload
netsh int tcp set global dca=enabled - enables Direct Access cache (should be supported by CPU, chipset and NIC)
To test the network bandwidth, it is recommended to use NTTTCP or iPerf.