How to keep Manufacturing IT failures from stopping production

Every plant wants the same simple thing: keep the line moving, all the time, without surprises. That’s easier said than done, because there are two very different ways production can grind to a halt. Sometimes it’s mechanical – a tool breaks, a conveyor jams, a part wears out past what it can handle. But just as often, it’s not the machine at all. It’s the systems running behind it: a Manufacturing Execution System (MES) instruction that never shows up, a historian that quietly stops logging, or a login screen that won’t let a reviewer sign off on a batch. Either way, the line stops, and money starts slipping away the moment it does.

The second kind of problem, on the IT and control-system side, needs a different fix. A failed server, network link, or storage system shouldn’t be able to bring production to a standstill. And when a failure does happen, the plant needs to recover fast enough that operators aren’t left waiting on an infrastructure rebuild.

First step: figure out which parts of the IT infrastructure the production workflow actually depends on.

Manufacturing IT infrastructure starts with the production workflow

Manufacturing IT infrastructure is the technology chain connecting business systems, plant applications, operators, and the equipment making product. An order enters through the Enterprise Resource Planning (ERP) system, a recipe or work instruction reaches MES, an operator confirms a step through a Human-Machine Interface (HMI), and a Supervisory Control and Data Acquisition (SCADA) system coordinates the equipment carrying it out. Each handoff depends on the previous one.

Downtime anywhere in that chain touches more than the immediate step. A stalled MES instruction delays the order behind it. A historian outage breaks the process record a quality reviewer needs later. An identity or network failure can block a quality release even when the physical batch is sitting there, finished and fine, waiting for a signature that a broken login screen won’t let anyone give.

This is why a service can look online while the production workflow it supports stays unusable. The MES web page may load, while the database behind it can’t be reached. The interface between MES and the historian may be down, leaving the two systems with different views of the batch state. None of that shows up as a single alarm. It shows up as a line that won’t move and an operator who isn’t sure why.

That dependency chain is what you need to protect when you design manufacturing IT infrastructure. Looking at individual servers or applications in isolation isn’t enough.

Important infrastructure layers

Industrial IT infrastructure is built from the same layers as any data center: compute, storage, virtualization, networking, identity, monitoring, and backup, plus whatever runs in the cloud or at the edge. What’s different is the plant conditions around those layers, and how directly each one maps to something that stops production.

Compute and virtualization host MES, SCADA supervisory components, historians, and plant databases. Storage holds process data, batch records, and recipes, usually with a local replica for failover and a separate copy for recovery. The plant network carries traffic between controllers, HMIs, and servers, and it needs to stay separate from the enterprise network that carries email, file shares, and other business traffic. Identity decides who can log in to MES, approve a quality release, or push a configuration change to a controller. Monitoring and backup tell you a failure happened and let you recover from it.

The boundary between Information Technology (IT) and Operational Technology (OT) sits inside this stack, and it matters because the two sides have different tolerances. IT can usually accept a reboot and a five-minute wait. OT often can’t.

IT/OT convergence changes how these systems are connected, but it doesn’t remove that tolerance gap.

Plant conditions make infrastructure management harder as well. Equipment may have to operate in cramped cabinets, heat and dust can shorten hardware life, power quality may be inconsistent, and legacy protocols may assume a flat and trusted network. Maintenance windows are often short, and many plants have little or no local IT staff available when something fails overnight.

For that reason, tie each infrastructure layer to a specific production dependency. A storage array may hold the batch record a quality review needs. A switch may be the only path between an HMI and the controller it’s trying to reach. That perspective makes it easier to spot which failures demand immediate failover and which can wait for standard recovery.

Which plant workloads can’t wait for the cloud?

Some applications need to keep running even when the connection to a central data center or the cloud is down. MES, SCADA, HMI, historians, plant databases, identity services for plant logins, machine-vision inspection, and parts of Quality Management System (QMS) or Laboratory Information Management System (LIMS) functionality typically fall into this group. They sit close to the physical process, and a Wide Area Network (WAN) outage can’t be allowed to stop them.

Other systems can tolerate delayed connectivity. ERP, Product Lifecycle Management (PLM), fleet-wide analytics, AI model training, reporting, and long-term archives generally don’t need millisecond response times. They can usually queue updates until connectivity returns.

The placement decision should be made workload by workload. Ask a few practical questions:

• How much latency can the application tolerate?
• Can it operate at all during a WAN outage, and for how long?
• How much data does it generate, and does that data need to stay local for volume or regulatory reasons?
• What Recovery Time Objective (RTO) and Recovery Point Objective (RPO) does the workflow behind it actually require?
• Does the application vendor support running the way you want to deploy it?

Once you’ve answered these for each workload, the local-versus-central split becomes much clearer.

Where should manufacturing workloads run?

On-premises, cloud, and hybrid designs each place workloads in different failure domains. The goal is to pick a model that matches what each production workflow actually needs.

On-premises keeps machine-facing services close to the equipment and reduces how much depends on the WAN. This matters for workloads that can’t pause during a connectivity outage.

Cloud platforms cut the amount of local hardware the plant needs to operate. At the same time, connectivity, identity services, provider availability, and data-location requirements all become part of the failure model. For some industries, where the production data sits and how it’s handled also bring in extra regulatory considerations.

Hybrid designs are the common middle ground in manufacturing: plant-critical systems stay local, and everything that can tolerate delayed synchronization runs centrally or in the cloud instead.

None of these models is automatically the right answer for every site. A plant with a reliable, well-provisioned WAN link can lean more on centralized or cloud services than a remote site on a single unreliable connection. The decision should follow what each workload actually needs, not a default architecture applied everywhere.

When you review a manufacturing environment, start with the production workflow, map its dependencies, and then decide where each component should run. That gives you a much clearer basis for designing high availability, backup, and disaster recovery in the next stages of the infrastructure plan.

Why different manufacturing sectors need different infrastructure

The same uptime target produces very different infrastructure priorities depending on what’s being made. A stalled line hits a discrete parts manufacturer differently than a pharmaceutical batch process, even when both call it “an hour of downtime.”

The difference becomes clearer when you look at the workflow behind the production line. Some environments depend on continuous process control, while others rely on tightly synchronized assembly, batch records, or high-volume data collection. Infrastructure resilience has to account for those differences.

The pattern behind the table matters more than any single row. Recovering the server is only step one. In an automotive plant, MES can come back online in minutes while the historian reconnects on its own schedule, and until it does, you’re running the line without the process data quality needs to release the shift’s output. It’s a different problem entirely in pharma: a server can be back up in ten minutes while the batch record shows a gap that triggers a full investigation before that batch can ship. The workflow recovery target isn’t the same as the server recovery target, and sector context is what tells you which one to design around.

How production IT fails

Most production outages aren’t caused by exotic infrastructure failures. Servers fail, storage paths drop, WAN links go down, power problems can take cooling systems offline with them. Databases run out of capacity. A routine update introduces a regression that never showed up in testing.

The less obvious dependencies are usually the ones that bite you. DNS, directory services, license managers, certificate services, and vendor remote-access tools often sit outside the main production application inventory, which means nobody notices them until something breaks. When one of them fails, the apps that depend on it stop working even though their own servers are perfectly healthy.

A DNS failure can knock several otherwise-healthy applications offline at once. An expired certificate can prevent a supervisory application from starting after a routine reboot, and you don’t find out until the next shift tries to log in.

Another scenario that comes up more than you’d think: a license server becomes unavailable during production. A scheduled Windows update reboots the standalone server during a shift, and the server is offline for only a short period, so the event may not trigger an obvious alert. Later, an operator’s MES session fails to authenticate or acquire a required license, and production stops while the team works backward through the dependency chain to find the cause.

These failures affect production in different ways, which is why treating every incident as a generic “server down” scenario leaves gaps in the recovery plan. Failing over a failed host can restore the VM and its storage. It can’t repair a missing DNS record, restore a license service that’s still offline, or fix an expired certificate. When you build the failure model, include the dependencies around the workload.

Designing resilient plant networks without blurring IT and OT

Network resilience starts with redundant paths. Depending on the site, you might add redundant plant switches, dual uplinks, diverse WAN connections, and a cellular backup link for critical locations.

SD-WAN shows up in most multi-site manufacturing rollouts because it can steer traffic across available links and keep connectivity alive when the primary path drops. If the WAN is unavailable, plant systems should still run locally where the workflow allows, with non-critical data queued for sync once connectivity comes back. That’s not optional; it’s how you keep a plant moving when the carrier’s fiber is cut somewhere between you and HQ.

None of that redundancy works if plant and enterprise traffic aren’t properly separated. An industrial DMZ sits between the OT and IT networks and controls what can cross that boundary, and IT/OT segmentation limits how far a problem on one side can spread to the other. Access control and Quality of Service (QoS) settings keep a large file transfer or a security scan on the enterprise side from starving MES or SCADA traffic on the plant side.

The same principle applies inside the plant. Storage replication, cluster management, backup traffic, and production traffic should have appropriately separated paths or controls. If they share a constrained link, a large backup or storage resync can compete with the traffic you need to keep the production workflow running, and you’ll feel it first in latency-sensitive workloads before you ever see an alert.

A dual-WAN, SD-WAN-backed design protects the plant from a connectivity failure. It doesn’t protect a local MES server from a hardware failure, and it doesn’t make the local storage highly available. Those need a different layer of the architecture.

HA, backup, and DR are three different jobs

High availability, synchronous replication, backup, and disaster recovery address different failure scenarios.

High availability keeps a workload available when a host, storage component, or local infrastructure path fails. With synchronous replication, the current data state is maintained on another node so the workload can restart there without waiting for a traditional restore. That’s the goal: minimize the gap between failure and resume.

There’s a catch that bites people who haven’t lived through it. Replication preserves the current state, including the bad changes. An accidental deletion, a corrupted write, a ransomware-encrypted file – any of those get replicated to the second node almost immediately, which means your two synchronized copies aren’t two independent recovery points. They’re the same current data in two locations, and you can lose the same way twice.

Backup covers a different class of failure. Manufacturing environments need recovery copies that sit apart from the production environment and stay clear of the credentials used to administer the HA cluster. If the same admin account that’s compromised in the cluster is also the one that runs the backups, the backups are toast too. Depending on the requirements, this can mean immutable, offline, or otherwise isolated copies.

Backup planning needs realistic RPOs and RTOs. More importantly, those targets need to be measured against the actual production workflow. A successful nightly backup confirms the data was copied. It doesn’t prove that the application will start cleanly after a restore, that its dependencies will be available, or that the full workflow can return to operation inside the required RTO. You only establish that through recovery testing.

Disaster recovery covers failures that exceed the local HA domain. A prolonged power outage, fire, flood, major network failure, or other site-level event can take both nodes in a two-node cluster offline at the same time. Site-level recovery has to be planned separately from host-level HA, because a cluster can deliver excellent local availability and still offer zero protection against the loss of the building that contains both nodes.

The recovery strategy should account for three different questions:

1. Can the workload survive a local infrastructure failure? HA addresses this.
2. Can we recover an earlier, known-good state? Backup addresses this.
3. Can production continue or be restored after the site itself is unavailable? Disaster recovery addresses this.

Exact requirements depend on the manufacturing sector, geography, and regulatory environment. NIST guidance, ISA/IEC 62443, NIS2, FDA requirements, and other frameworks may apply depending on the organization and the systems involved. Where regulatory requirements apply, documented procedures and tested recovery evidence are generally as important as the redundant infrastructure itself.

A two-node cluster is one layer of the recovery strategy, not the whole thing. The next step is to connect that architecture to the specific infrastructure components that determine whether a plant can actually fail over and recover under production load.

What happens when a production host fails?

Picture two servers, each hosting a share of the plant’s critical VMs, synchronously replicating committed storage writes between them. When one server fails, the cluster has to notice quickly, decide which node is authoritative, and bring the affected VMs back up on the surviving node using the storage copy it already has.

Failure detection relies on heartbeat paths between the nodes, so the surviving node can tell a genuine failure from a temporary blip. Health checks should verify more than whether a server answers a ping. The real question is whether the services supporting production are actually available. Once the failure is confirmed, ownership of the affected storage moves to the surviving node and the VMs restart there.

That surviving node needs to be sized to carry the priority workload on its own. A second server sitting idle isn’t enough. It has to provide enough CPU, memory, storage performance, and network capacity to keep the production workflow inside its required performance limits.

HA doesn’t mean zero disruption. VMs need time to restart, applications need to reconnect, and some services may require additional recovery steps. A well-designed cluster can cut the interruption to seconds or minutes, but the exact recovery time depends on the workload and the failure scenario.

The architecture is built for a specific class of failures: a host, storage component, or local infrastructure path becoming unavailable while the other node stays operational. It doesn’t provide universal zero-RPO or zero-RTO protection, and it can’t cover every type of failure. StarWind Virtual SAN and StarWind HCI Appliance both implement this two-node model. The sizing, quorum, network, and recovery principles below apply to either approach.

What makes failover work under plant conditions

Whether HA holds up during an actual incident depends on details that are easy to overlook during normal operation. A cluster can look perfectly healthy in a dashboard and still have a serious weakness that surfaces only when a network path, power source, or storage component partially fails.

The failure-domain design is where this matters.

When cluster nodes lose contact

What happens when a cluster node is fine, but its connection to its partner drops? A pulled cable or a bad switch port can leave two nodes running blind. If both decide they’re the lone survivor, they both continue writing data independently. The storage splits, and you end up with two competing versions of the truth. Preventing this “split-brain” condition is the hardest job of any HA architecture.

To stop it, most platforms require an external tiebreaker. Solutions like VMware vSAN and Nutanix use a witness VM to cast the deciding vote. In a perfect design, this witness sits in a separate failure domain. In reality, plant IT teams often try to save a rack unit by parking the witness next to one of the primary nodes, or plugging it into the same UPS. When that UPS fails, the cluster loses the primary node and the witness at the same second. The surviving node is left without a quorum, and production stops.

Some implementations take a different approach. StarWind, for example, uses a dedicated heartbeat connection between the nodes rather than requiring a separate witness appliance or VM. The trade-off and the underlying requirement are the same: two nodes must never be allowed to believe that they independently own and can modify the same storage at the same time.

When you review a two-node design, look at the complete failure path. Ask what happens if the replication link fails, if the management network fails, if a switch disappears, or if one power circuit goes down. A cluster should have a deterministic answer for each scenario.

Resynchronization without overwhelming production I/O

Synchronous replication has a built-in trade-off. Every write waits for both nodes to confirm it before it’s acknowledged, which adds latency compared to writing locally and moving on. That trade-off buys you a current copy on both sides at the cost of a small delay on every write.

The heavier load shows up after a repaired node rejoins the cluster and needs to resynchronize, copying everything it missed while it was down. That traffic competes with normal production I/O, any failover activity still settling out, and scheduled backup windows, all potentially at the same time. Most platforms, StarWind included, let you throttle or schedule resync traffic so it doesn’t flatten production I/O during the day, but that only helps if someone actually configures it.

Left at defaults, a node that’s been offline for several hours may have a substantial amount of data to synchronize. If the cluster was sized only for average daytime utilization, the additional resynchronization load can expose a capacity problem precisely when the environment is already operating in a degraded state.

For that reason, test the rebuilding state as part of HA validation. Don’t measure only normal production performance and failover time. Measure what happens when a node returns and the cluster is rebuilding its replica while production remains active. (This is the part teams usually skip, and the rebuild runs straight into the next business day.)

Removing shared network and power failure points

Redundancy only protects against a failure if the redundant components are actually independent. Two NICs connected to the same switch don’t give the same protection as two NICs connected to independent switches. Two power supplies connected to the same circuit don’t give the same protection as supplies connected to separate power paths.

The same principle applies to the infrastructure between the servers and their external dependencies.

A cluster can have redundant NICs, switches, UPS units, and power circuits. It’s still vulnerable if both paths eventually converge on the same switch, distribution panel, power circuit, or cable route.

When you’re reviewing a plant cluster, trace each network and power path back to its source. Where do the paths converge? What’s the single failure that could take both sides offline? The exercise often surfaces dependencies that don’t appear in the cluster configuration itself – shared PDUs on the same phase, two fiber runs through the same conduit, that kind of thing.

Sizing the plant edge for machine vision and AI

Manufacturing edge computing moves data processing into or close to the plant instead of sending every operation to a remote data center and waiting for a response.

Machine vision inspection, predictive maintenance, sensor anomaly detection, telemetry aggregation, digital work instructions, and operator-support applications are the workloads pushing up demand for local compute.

These workloads add a sizing layer on top of what MES and SCADA need.

Machine vision and AI inference can eat substantial CPU and memory. Image plus sensor data creates real storage pressure. Your network has to absorb the movement of that data without it stepping on production traffic. Depending on the workload, a GPU or other accelerator may be required.

Power and cooling need to be in the design from day one. Extra compute at the edge pushes both requirements up, especially in plants where the existing equipment room is already running close to its environmental limits.

Software lifecycle matters too. AI models need to be updated, tested, and deployed without disrupting production.

The deployment process should be designed so model updates can be staged and rolled back without taking equipment offline – usually meaning a parallel environment that mirrors production closely enough to catch regressions before they hit the line, and a rollback path that’s been tested under load rather than on paper.

Data retention is the other sizing variable you can’t ignore.

How long do inspection images, sensor telemetry, and the rest of the edge-generated data need to stay available before they’re archived or dropped?

Holding more history costs storage, but it gives you material for investigating quality issues or retraining models later.

How to choose infrastructure for a manufacturing site

Start with the production workflows that need to stay available.

For each critical workflow, define the RTO and RPO the business can tolerate.

Then weigh the factors that shape the design:

• WAN reliability at the site
• Number of plants and locations to deploy
• Existing server and storage hardware
• Internal infrastructure expertise
• Application vendor support for the proposed deployment model
• Production data growth
• Backup and recovery requirements
• Remote management and support requirements
• Expected machine vision and AI workloads over the next one to two years.

That same evaluation tells you whether existing hardware should be reused or replaced.

Existing servers paired with software-defined storage, such as the StarWind Virtual SAN approach described above, make sense when you’ve already got capable hardware and the in-house expertise to run it. That’s most practical for a single plant or a small footprint.

A pre-integrated platform such as StarWind HCI Appliance fits better when standardization across multiple sites matters, when there’s no existing hardware worth reusing, or when a single vendor relationship for infrastructure and support simplifies operations.

Your choice should reflect how the plant is actually operated. Hardware flexibility pays off at one site. Standardized deployment and centralized support often matter more across dozens of locations.

Plan around the process, not the server

Every section above comes back to the same point: plan around the production process that has to keep running, and let the server, network, storage, and WAN design follow from that.

A cluster that protects the wrong workload, or protects the right one without accounting for what happens during resynchronization, doesn’t actually solve the problem it was built for.

HA, backup, and disaster recovery each cover a different failure. A design that treats them as interchangeable leaves gaps you don’t notice until the wrong one fires.

FAQ

Can a plant continue operating during a WAN outage?

Yes, as long as critical workloads – MES, SCADA, HMI, historian, and plant identity services – run locally. Systems like ERP and analytics can queue updates until connectivity comes back.

Can MES and SCADA run on a two-node virtualized cluster?

Yes. A two-node HA cluster with synchronous replication protects MES and SCADA from a single server or storage failure. The surviving node has to be sized to carry the priority workload alone, with replication, management, and production traffic properly separated.

Does synchronous replication replace backup?

No. Replication also copies deletions, corruption, and ransomware to the second node – that’s the part people miss when they decide synchronous replication is good enough and skip the backup tier entirely. Backup gives you separate recovery points and should use isolated, immutable, or offline copies. Test recovery by restoring the application and its dependencies, not by checking that the backup job completed.