"Can we segment the network without stopping production?" We are asked this in almost every OT conversation, usually by a plant or facility manager who has already lived through one security project that caused an unplanned stop. Our answer is yes. But the reason most segmentation projects stall has little to do with firewalls, and a lot to do with what happens between the approved design and the first rule that blocks something.
Our earlier piece on what actually works in IT/OT segmentation covers the target architecture. This one answers the questions we get about the harder part: getting there on a network that cannot go down.
"Our network is simple. Why would anything break?"
Because it is rarely as simple as the drawing. On paper, a control network carries a handful of known flows: HMI to PLC, PLC to historian, engineering workstation to everything. A site that has grown over ten or fifteen years carries traffic nobody designed and nobody remembers.
- Discovery and broadcast traffic. BACnet Who-Is, EtherNet/IP and PROFINET discovery, NetBIOS, vendor-specific announcements. Some applications break quietly when these stop crossing a boundary.
- OPC Classic. DCOM negotiates dynamic ports. Firewall it without understanding it and the historian keeps its connection until the next restart, then never gets it back.
- Dependencies that only appear occasionally. A licence server checked once a week. A report generated at month end. A redundancy failover that has not happened since commissioning. These hide in a two-week traffic capture and surface three months after go-live.
- Hard-coded addressing. Controllers with fixed IPs and gateways baked into the logic, where any change means a program download, and the person who wrote the program left the integrator years ago.
- Things that are not on the drawing. A cellular modem a vendor installed for support. A second network card on an engineering laptop. A switch in a cabinet that quietly bridges two "separate" networks.
None of this comes up in the design review. All of it comes up during cutover.
"How long do we need to watch the network before we change anything?"
Longer than most project plans allow. The single biggest factor in a quiet cutover is how long the network was observed before anything changed. Passive monitoring across a full operating cycle, including a maintenance shutdown, a month end and ideally a seasonal changeover, surfaces most of the surprises above. Two weeks of capture will not.
This is also where the asset inventory earns its keep. Every flow should map to a known asset with a known owner. Flows nobody can explain are not noise. Either they are a dependency you are about to break, or they are something that should not be there, and you want to know which before the firewall decides for you.
We also always ask to speak with the people who run the process, not only the people who maintain the network. The shift supervisor knows that one line's HMI freezes whenever the packaging server reboots. That fact is in no document.
"Where should we start?"
It is tempting to start where the network is cleanest. That gives an early win and very little risk reduction. We prefer an order driven by consequence and exposure together:
- The IT/OT boundary first. Most realistic attack paths into a plant or building cross it, and the flows across it are usually fewer and better understood than the flows inside OT. Tightening the boundary and moving remote access into a proper DMZ gives the biggest reduction for the least operational risk. Our article on vendor remote access covers that piece.
- Then the areas whose failure would hurt most. The OT risk assessment should tell you which they are. Separating a high-consequence process area from the rest of the site protects it from whatever happens next door.
- Safety systems stay separate throughout. If a safety instrumented system shares a network with basic process control, fixing that is a project of its own, owned by process safety engineering. It should not be a line item in a segmentation rollout.
The IEC 62443 zones and conduits model is the right frame for where you want to end up. It says nothing about the order of work, and in our experience the order is what decides whether production notices.
"Will we have to re-address our controllers?"
Clients dread this question for good reason. Re-IPing controllers is the most disruptive part of most segmentation designs. Firewalls deployed in transparent or bridged mode can enforce policy between groups of devices on the same subnet without touching their addresses or gateways. It is not always the right end state, but it often lets you get enforcement in place now and leave re-addressing to a planned upgrade.
"What if a rule blocks something we need?"
Then it should not have been enforcing yet. New rules should first run in a mode that logs what they would have blocked, long enough to cover the occasional dependencies described above. Only when that log stays clean does a rule start enforcing. It takes patience, and skipping it is how most outages in these projects happen.
The other half of the answer is rollback. If the only person who can revert a change is the security engineer who made it, and that engineer is not on site at 3 a.m., there is no rollback. In our view, the fastest way to earn an operations team's trust is a rollback they control and have practised.
"What happens if the firewall itself fails?"
A good question, and one that is asked too rarely. An inline firewall becomes part of the control path. What happens when it loses power, overheats in a cabinet with no cooling, or needs a firmware update? Fail-open or fail-closed is a process decision, not a security one. It should be made with operations and written down before installation. Industrial-rated hardware and redundant pairs on critical conduits are not luxuries.
"Who owns the rules once the project is over?"
This is the question we push clients to answer at the start, because it decides whether the work survives. If IT owns the rules, every new instrument or vendor connection becomes a ticket in a queue that does not understand urgency on the plant floor. If OT owns them with no security oversight, the rule base drifts back toward "allow any" within a couple of years. Shared ownership with a clear approval path is the arrangement that tends to hold.
Each enforcement step is also a change to a production system and should go through the same management of change as any other. That means working inside maintenance windows, which are scarce and planned months ahead. We made the same point about patching in OT patch management: book security work into the windows early, or mechanical work will push it out.
"How do we know the project is on track?"
You can tell without reading the design documents:
- Every rule has a named business reason and an owner on the operations side.
- The number of broad "allow any" rules goes down over time instead of quietly up.
- Operations raise new connectivity needs before plugging something in, because asking is faster than working around.
- Firewall logs feed monitoring that someone actually reviews, and blocked traffic gets investigated rather than allowed on reflex.
- When an incident response exercise asks "can we isolate this area without stopping the line?", the answer is yes, and people know which conduit to close.
That last one is the real payoff. Segmentation is not only about keeping attackers out. It is what lets you contain a problem in one area while the rest of the site keeps running.
What we tell clients before they start
The technology is rarely the obstacle. Projects stall when the security team cannot explain a rule in terms an operations manager cares about, or when one early outage turns the site against the whole effort. Both are avoidable. They come down to understanding the process before touching the network, and moving at the pace production can absorb rather than the pace the project plan wants.