MIMO SAN NETWORK SETUP

clayton@mcc911.org · by **clayton@mcc911.org** » Wed Mar 09, 2011 11:14 pm

Hello guys i need some help
i cant wrap my head around this.
i am seting up starwind in a 2 node HA with 3 ESXI boxes
all five boxes have 4 Gb ports for san traffic

i have defined 4 subnets
192.168.100.0 ,101.0 ,102.0 and 103.0

each of the five computers will have 1 port in each subnet

my question is,, i have 2 managed switches in a stack for the SAN
how should i do the cables and vlans
right now there is 1 port in a vlan for switch managment and all the the others in a second vlan
there will only be san traffic on these 2 switches,

should i make 4 vlans for each of the for subnets ? is that really needed?
how would you plug the cables up ? put all off sub net one on switch one and all of sub net 2 one switch two and all of sub net 3 on switch one and all of sub net 4 on switch 2

or all four ports of node one on switch one and all four ports of node 2 on switch 2 and the spread the four ports of the esxi boxes accross both??
and id you did it that way if you used vlans you would have to set vlans accross both swiches....
there just seems to be 6 ways to sunday here please advise
thank you
Clayton

Anatoly (staff) · by **Anatoly (staff)** » Thu Mar 10, 2011 11:20 am

Hello Clayton

I`ve attached recommended scheme for your system

Please note that interfaces between StarWind Servers are used for synchronazation channel and used to be bounded and connected directly. Also switches are connected in order to increase fault-tolerance of your system.

clayton@mcc911.org · by **clayton@mcc911.org** » Thu Mar 10, 2011 5:15 pm

Anatoly

thank you for the help.
the 4 ports I listed are just for San Traffic
each server has 2 four port cards plus 4 ports on board but your diagram will help

4 port to each switch
4 port to each other for the replication link
and 4 onbord ports for managment ,lan connetions, ip-kvm ect.

but what about the Vlans on the switches do i need them

anton (staff) · by **anton (staff)** » Thu Mar 10, 2011 8:53 pm

You don't really need VLAN thing as whole iSCSI network should be run outside DMZ. I would go for dedicated SAN fabric / hardware however. Mostly b/c of performance.

clayton@mcc911.org · by **clayton@mcc911.org** » Fri Mar 11, 2011 2:42 am

ok thanks
this is want i set up today before i saw your post
so far i have just loaded server on both and got all the swiches set up
so i dont know if it will work but unless i am missing somethign it should work well.

Anatoly (staff) · by **Anatoly (staff)** » Fri Mar 11, 2011 9:11 am

I think that me and Anton gave you pretty detailed recommendation, but your plan will work too. Please note that you have no need to use four vlans to connect to SANs.
To learn more please read SAN configuration guide from VMware pubs. Here is for ESX 4.0 and ESX 4.1

mkaishar · by **mkaishar** » Sat Mar 12, 2011 1:48 am

I think 4 VLANs for the SAN environment is overkill and it will be a pain to manage in the future

Our environment is split into a LAN VLAN and a SAN VLAN using the same switches. We don't have Starwind HA at this time, but the settings below should work.

I would change your Sync network to something completely different from your SAN network, maybe it's just me, but using 192.168.xxx for everything just gives me more opportunities for mistakes.

Use a 10.xx or a 172.xx

I like using 10.xx network because they are much easier to type

I know there are recommendations to split physical switches for performance, but that is a waste of ports & money.

Here is what I use through Putty and SSH into on each ESXi server to create the config, change IPs as needed.
I enable everything through console, I guess you can run it through PowerCLI, I just haven't had the time to learn it all yet.

Code: Select all: esxcfg-vswitch -a vSwitch1 esxcfg-vswitch -m 9000 vSwitch1 esxcfg-vswitch -A iSCSI0 vSwitch1 esxcfg-vmknic -a -i 10.0.0.20 -n 255.255.255.0 -m 9000 iSCSI0 esxcfg-vswitch -A iSCSI1 vSwitch1 esxcfg-vmknic -a -i 10.0.1.20 -n 255.255.255.0 -m 9000 iSCSI1 esxcfg-vswitch -A iSCSI2 vSwitch1 esxcfg-vmknic -a -i 10.0.2.20 -n 255.255.255.0 -m 9000 iSCSI2 esxcfg-vswitch -A iSCSI3 vSwitch1 esxcfg-vmknic -a -i 10.0.3.20 -n 255.255.255.0 -m 9000 iSCSI3 esxcfg-vswitch -A iSCSI4 vSwitch1 esxcfg-vmknic -a -i 10.0.0.120 -n 255.255.255.0 -m 9000 iSCSI4 esxcfg-vswitch -A iSCSI5 vSwitch1 esxcfg-vmknic -a -i 10.0.1.120 -n 255.255.255.0 -m 9000 iSCSI5 esxcfg-vswitch -A iSCSI6 vSwitch1 esxcfg-vmknic -a -i 10.0.2.120 -n 255.255.255.0 -m 9000 iSCSI6 esxcfg-vswitch -A iSCSI7 vSwitch1 esxcfg-vmknic -a -i 10.0.3.120 -n 255.255.255.0 -m 9000 iSCSI7 esxcfg-vswitch -L vmnic4 vSwitch1 esxcfg-vswitch -L vmnic5 vSwitch1 esxcfg-vswitch -L vmnic6 vSwitch1 esxcfg-vswitch -L vmnic7 vSwitch1 esxcfg-vswitch -p iSCSI0 -N vmnic5 vSwitch1 esxcfg-vswitch -p iSCSI1 -N vmnic5 vSwitch1 esxcfg-vswitch -p iSCSI0 -N vmnic6 vSwitch1 esxcfg-vswitch -p iSCSI1 -N vmnic6 vSwitch1 esxcfg-vswitch -p iSCSI0 -N vmnic7 vSwitch1 esxcfg-vswitch -p iSCSI1 -N vmnic7 vSwitch1 esxcfg-vswitch -p iSCSI2 -N vmnic4 vSwitch1 esxcfg-vswitch -p iSCSI3 -N vmnic4 vSwitch1 esxcfg-vswitch -p iSCSI2 -N vmnic6 vSwitch1 esxcfg-vswitch -p iSCSI3 -N vmnic6 vSwitch1 esxcfg-vswitch -p iSCSI2 -N vmnic7 vSwitch1 esxcfg-vswitch -p iSCSI3 -N vmnic7 vSwitch1 esxcfg-vswitch -p iSCSI4 -N vmnic4 vSwitch1 esxcfg-vswitch -p iSCSI5 -N vmnic4 vSwitch1 esxcfg-vswitch -p iSCSI4 -N vmnic5 vSwitch1 esxcfg-vswitch -p iSCSI5 -N vmnic5 vSwitch1 esxcfg-vswitch -p iSCSI4 -N vmnic7 vSwitch1 esxcfg-vswitch -p iSCSI5 -N vmnic7 vSwitch1 esxcfg-vswitch -p iSCSI6 -N vmnic4 vSwitch1 esxcfg-vswitch -p iSCSI7 -N vmnic4 vSwitch1 esxcfg-vswitch -p iSCSI6 -N vmnic5 vSwitch1 esxcfg-vswitch -p iSCSI7 -N vmnic5 vSwitch1 esxcfg-vswitch -p iSCSI6 -N vmnic6 vSwitch1 esxcfg-vswitch -p iSCSI7 -N vmnic6 vSwitch1 esxcli swiscsi nic add -n vmk1 -d vmhba38 esxcli swiscsi nic add -n vmk2 -d vmhba38 esxcli swiscsi nic add -n vmk3 -d vmhba38 esxcli swiscsi nic add -n vmk4 -d vmhba38 esxcli swiscsi nic add -n vmk5 -d vmhba38 esxcli swiscsi nic add -n vmk6 -d vmhba38 esxcli swiscsi nic add -n vmk7 -d vmhba38 esxcli swiscsi nic add -n vmk8 -d vmhba38

I put this script in /etc/rc.local for MPIO

Code: Select all: # Fixing iSCSI MPIO througput limited to 1G. esxcli nmp device list | grep ^eui | while read device ; do esxcli nmp device setpolicy --psp VMW_PSP_RR --device ${device} esxcli nmp roundrobin setconfig --type "iops" --iops 3 --device ${device} done

If your SAN NIC's are Intel's and support VMDQ, you should enable:

http://kb.vmware.com/selfservice/micros ... 0160190099

http://kb.vmware.com/selfservice/micros ... Id=1004593

Code: Select all: esxcfg-module -e igb esxcfg-module -s "IntMode=2,2,2,2 VMDQ=4,4,4,4" igb

I also use these recommended settings from Starwind:

starwind-f5/recommended-settings-for-esx-iscsi-initiator-t2296.html

starwind-f5/recommended-tcp-settings-t2293.html

anton (staff) · by **anton (staff)** » Sun Mar 13, 2011 4:48 pm

1) VLANs are definite overkill. Consensus!

2) "Use a 10.xx or a 172.xx" (c) ... Did you actually mean 192.xx?

mkaishar · by **mkaishar** » Mon Mar 14, 2011 2:35 am

I meant for sync traffic use 10.xx or 172.16.xx which would be completely different from the OP's 192.168.xxx SAN network

Anatoly (staff) · by **Anatoly (staff)** » Mon Mar 14, 2011 8:44 am

Oh, then yes, sure, then we`ve got consensus at this point too

clayton@mcc911.org · by **clayton@mcc911.org** » Mon Mar 14, 2011 3:05 pm

thanks guys good tips!!
i use 192.x because its jsut easy for me. my lan is a 172. and all my wan links and hops are 10. so it just easy for me to know all my san suff is 192.168. 1~5.x

Anatoly (staff) · by **Anatoly (staff)** » Mon Mar 14, 2011 3:31 pm

You`re very welcome. Please keep us updated.

MIMO SAN NETWORK SETUP

MIMO SAN NETWORK SETUP

Re: MIMO SAN NETWORK SETUP

Re: MIMO SAN NETWORK SETUP

Re: MIMO SAN NETWORK SETUP

Re: MIMO SAN NETWORK SETUP

Re: MIMO SAN NETWORK SETUP

Re: MIMO SAN NETWORK SETUP

Re: MIMO SAN NETWORK SETUP

Re: MIMO SAN NETWORK SETUP

Re: MIMO SAN NETWORK SETUP

Re: MIMO SAN NETWORK SETUP

Re: MIMO SAN NETWORK SETUP

Who is online