HA Server dirty shutdown with Write-Back Cache?

[camealy]

In the current version of 5.5 if you have a decent size write-back cache of say 1GB for an HA target and one of the HA servers crashes, what is the result? I understand that the each partner should have redundant power supplies, different UPS's, etc. But even in a perfect world a motherboard can go up in smoke...

Does the HA set protect you in any way from the loss of data in the cache that hadn't been written to disk?

i.e. Will I corrupt my SQL and Exchange Virtual Machines running on on the HA set?


Thanks!

Kurt

[anton (staff)]

The whole idea of having HA is to have redundancy for everything: CPU horsepower, RAM, power supply, network wiring, switches etc etc etc. Including write-back cache of course! So caches on both HA nodes are distributed and synchronized. If one HA node is down (maintenance, PSU failure, nuclear missle blast or whatever you can imagine) other healthy node immediately starts to write cache to the disk and STOPS caching incoming requests. So your multigigabyte cache is expected to be saved to the constant media pretty fast (keeping in mind modern I/O subsystem performance). When done you still have HA system in service (one node is alive) so can bring back second node, initiate disk sync sequence and wait for some time for HA storage cluster becoming healthy completely. During part-time failure combined performance is going to suffer (obviously as there's no write back cache any more, all writes go directly to disk and all reads don't bypass disk, 100% cache miss scenario) but system is fully working. When HA cluster gets recovered, cache is auto-enabled automagically and systems starts to blow the wind at the full speed.

In the current version of 5.5 if you have a decent size write-back cache of say 1GB for an HA target and one of the HA servers crashes, what is the result? I understand that the each partner should have redundant power supplies, different UPS's, etc. But even in a perfect world a motherboard can go up in smoke...

Does the HA set protect you in any way from the loss of data in the cache that hadn't been written to disk?

i.e. Will I corrupt my SQL and Exchange Virtual Machines running on on the HA set?


Thanks!

Kurt

[camealy]

OK, great.

One small feature request would be the ability to set throttle percentages or priority on the full-sync. In heavy testing with bonded 1GB links, I notice that read/write performance isn't even enough to keep a virtual machine responding during a full-sync.

I don't know which would be easier to implement; the ability to fast-sync after a clean shutdown of both nodes (extended power outage), or the ability to throttle the full-sync rebuild so servers could at least be at a functional level after things come back online.

Thanks!

Kurt

[anton (staff)]

Already on the schedule.

OK, great.

One small feature request would be the ability to set throttle percentages or priority on the full-sync. In heavy testing with bonded 1GB links, I notice that read/write performance isn't even enough to keep a virtual machine responding during a full-sync.

I don't know which would be easier to implement; the ability to fast-sync after a clean shutdown of both nodes (extended power outage), or the ability to throttle the full-sync rebuild so servers could at least be at a functional level after things come back online.

Thanks!

Kurt

[camealy]

That is good... Which path is StarWind heading down, throttling settings for full-sync, or fast-sync after clean shutdown?

Also, if the cache has writes replicated between partners, is that happening over the sync channel? i.e. Is every Host writing to both HA partners over the SAN channel, or is the Host only writing to one side and that HA partner writes to the other over the SYNC channel?
The reason I ask, is with 1GB links easily being saturated and 10GB hardware still on the expensive side, I wonder if having 1 SYNC link-aggregate pair per target would improve HA speed.

Thanks!

Kurt

[Max (staff)]

StarWind will be able to do both, you'll have the ability to change the sync priority.
By the way, the clean shutdown does not involve any sync if it is really clear:
If you disconnect the targets from the clients, resynchronize the HA(if necessary) and remove it from the console then you will be able to recreate it without synchronizing after the servers are back online.
Regarding the sync - this is a good idea, having it faster than GbE will make your HA faster.
The data is written to nodes in a round robin mode and then synchronized between them

[anton (staff)]

Pair of 10 GbE cards cross-connected to each other (w/o a switch) is not going to break the bank. But HA performance should be boosted dramatically.

That is good... Which path is StarWind heading down, throttling settings for full-sync, or fast-sync after clean shutdown?

Also, if the cache has writes replicated between partners, is that happening over the sync channel? i.e. Is every Host writing to both HA partners over the SAN channel, or is the Host only writing to one side and that HA partner writes to the other over the SYNC channel?
The reason I ask, is with 1GB links easily being saturated and 10GB hardware still on the expensive side, I wonder if having 1 SYNC link-aggregate pair per target would improve HA speed.

Thanks!

Kurt

[camealy]

I understand the remove targets and re-add, but that is just tough to know when the power is going to fail

I already have bonded 1GB pairs for the Sync channel, but I was curious if there would be a noticeable improvement if bonded pairs for SYNC were dedicated and different for each .img The problem comes in how link-aggregations don't send a single TCP stream over both links for 2GB of bandwidth.

Thanks,

Kurt

[Max (staff)]

I understand the remove targets and re-add, but that is just tough to know when the power is going to fail

Oh, i thought you mean a planned power outage:) because we can't talk about a clean shutdown for both servers in case of an outage, it will never be too much to check the disks after such thing occurs. It may happen that one server is 1-2 bytes ahead the partner.

[camealy]

Right, but you would think that if UPS's were configured for 5 minute staggered clean shutdowns, (using the UPS software) the fast sync logs could track that server 2 went down first and a few minutes later server 1 went down and the change rate was low enough (due to the ups software first shutting down the vm's before the SAN partners) that a fast sync would be fine after restart. I realize that would also require some sort of tracking of who went down last in the software so it would know which side to declare out of sync.

[Max (staff)]

Right, but you would think that if UPS's were configured for 5 minute staggered clean shutdowns, (using the UPS software) the fast sync logs could track that server 2 went down first and a few minutes later server 1 went down and the change rate was low enough (due to the ups software first shutting down the vm's before the SAN partners) that a fast sync would be fine after restart. I realize that would also require some sort of tracking of who went down last in the software so it would know which side to declare out of sync.

then your UPS should first switch off the clients and the networking.

[camealy]

Pair of 10 GbE cards cross-connected to each other (w/o a switch) is not going to break the bank. But HA performance should be boosted dramatically.

Can you cross connect dual-port 10GbE cards in a static-link aggregation? I thought a switch was needed for that?

Thanks,

Kurt

[anton (staff)]

You don't need switch to enable link aggregation.

Pair of 10 GbE cards cross-connected to each other (w/o a switch) is not going to break the bank. But HA performance should be boosted dramatically.

Can you cross connect dual-port 10GbE cards in a static-link aggregation? I thought a switch was needed for that?

Thanks,

Kurt

[camealy]

Any chance 5.6 has the sync priority?

Already on the schedule.

OK, great.

One small feature request would be the ability to set throttle percentages or priority on the full-sync. In heavy testing with bonded 1GB links, I notice that read/write performance isn't even enough to keep a virtual machine responding during a full-sync.

I don't know which would be easier to implement; the ability to fast-sync after a clean shutdown of both nodes (extended power outage), or the ability to throttle the full-sync rebuild so servers could at least be at a functional level after things come back online.

Thanks!

Kurt

[@ziz (staff)]

Any chance 5.6 has the sync priority?

Already on the schedule.

OK, great.

One small feature request would be the ability to set throttle percentages or priority on the full-sync. In heavy testing with bonded 1GB links, I notice that read/write performance isn't even enough to keep a virtual machine responding during a full-sync.

I don't know which would be easier to implement; the ability to fast-sync after a clean shutdown of both nodes (extended power outage), or the ability to throttle the full-sync rebuild so servers could at least be at a functional level after things come back online.

Thanks!

Kurt

It's already available in v5.6 which we released few days ago.