Cyber security risks must be evaluated at all levels of IT, both enterprise and production, especially when a company is moving towards Smart Manufacturing.
One of the cornerstones of Smart Manufacturing is OT-IT convergence. Information technology (IT) systems primarily work with data, and operational technology (OT) systems monitor and adjust events, processes and devices in industrial applications.
OT-IT convergence is the connection of shop floor systems (ISA 95 Layer 0-2) like sensors, actuators or PLCs, with the office floor (ISA Layer 4) and systems like ERP, PLM or CRM, QM. The ISA Layer 3 systems like MES, LIMS, WMS mix both shop floor and office floor, and therefore build the bridge for OT-IT convergence.
To be successful with OT-IT convergence, the two IT disciplines, enterprise or corporate IT and production IT, have to come together.
Enterprise IT vs production IT
Until now enterprise IT has typically taken care of all of the IT systems situated in ISA Layer 4. They have usually specialized in:
- Administering operating systems, both server and client side
- Cyber security
- Network administration
- User administration
- Network architecture
- Hardware architecture, mostly for clusters, data warehouses, cloud computing or VM farms
- Administration of enterprise applications like ERP, QM, PLM etc.
- B2B communication
On the other hand production IT has focused more on ISA Layer 0-2 and brings knowledge in automation to the table:
- PLC & DCS architecture and programs
- Production networks/bus systems
- Shop floor computers/industry PCs
- Robotic architecture and programs
- M2M communications
- Process Control Architecture
The most obvious difference here is the lack of focus on cyber security in the production IT.
Cyber security in production IT
In the most cases the lack of cyber security in production was okay – there was no connectivity to the more vulnerable enterprise networks. An enterprise network is:
- Always connected to the internet.
- Connects many nodes (both client & server nodes) which form the enterprise architecture.
This means that there are many attack vectors, so trying to keep a company’s enterprise architecture from harm through regular patching, updating of white listing and black listing (antivirus applications) applications, network monitoring and adjusting firewall rules are every-day tasks in enterprise IT.
Production IT on the other hand, until recently, rarely had to worry about cyber security. Their systems were typically not connected to the enterprise networks and were therefore safe from outside attacks. In some cases, even if a connection existed, there was little concern about production systems, because it was assumed that an attack could not and would not reach so deep.
Attacks at the production level
The danger of an attack through infected USB drives was often perceived as more real than an attack originating in the enterprise network. And attacks like Stuxnet seemed to prove the truth of this belief.
But this risk could be handled through work discipline, and didn’t require additional knowledge.
Little was done in terms of an active cyber security in a production environment.
This all started to became more urgent with the introduction of systems, like MES, which are sitting in ISA Layer 3 and by default, have both an enterprise connection and a connection to layer 2 systems. This makes it possible for cyber attacks to reach down right onto the shop floor easily.
This meant that the now vulnerable production networks had to be included in the cyber security strategy.
Securing IT systems
Both the enterprise and production IT staff had to start working together, often leading to friction and obstruction because of their different focus and views.
Enterprise IT must ensure that all systems are up to date, both in terms of the applications as well as patches to minimize the vulnerabilities that cyber criminals can exploit. White listing, black listing or both are maintained on each system. Users are heavily restricted in terms terms of what can they do, where they have access, and passwords valid periods.
These are just a few of the common enterprise IT practices.
Production software and hardware is usually out of date
On the other hand, production IT often uses hardware and software which is either near its end of life, or even past this stage. New operating systems or hardware often don’t support the versions of the application required to operate the production process. The turnover rate of production hardware is as low as for the actual machines themselves. New hardware and software is introduced only when the machine is upgraded or a new machine is purchased. Companies have therefore many legacy systems to take care of.
Bad security practices in production IT
The recent Wanacry ransomware attack showed how vulnerable old hardware and software are, and that massive damage can occur. If this would hit for example a DCS or SCADA system, the plant would be blinded, and, in the worst case, may cause damage to the environment and human lives through uncontrolled reactions.
Insecure user account practices
When it comes to user handling, it’s normal to use group or superuser accounts, because it makes the shop floor “easier” to handle. In some cases the username and password are even in clear text, as paper notes lying around the workstations.
Some older systems have the issue that login information are not encrypted when a user logs in from a client. You can easily discover valid credentials through network sniffing. Accounts and passwords tend to not expire, and old accounts are not deleted regularly. All of this makes it easy for someone to exploit these systems.
Security through obscurity is not secure
Concerns are brushed off because ‘nobody knows the system so they can’t do anything with it, even if they get access’. But security through obscurity is no security. Many tools are available that allow intruders to observe what is done, for example, by recording keyboard strokes or even remotely watching what the user does. Training programs are still available for many legacy systems, so attackers can learn how to use it.
No separation between systems
When it comes to cybersecurity, you can often find a firewall between Layer 3 (MES) and Layer 2 (SCADA, PLC, DCS). But for M2M communication there is little to no security in use. It is easy to employ connected devices for a cyberattack, as done in October 2016 where cameras and NVRs were used to mount a DDoS attack on several services like Twitter and NetFlix.
Lack of updates creates security vulnerabilities
It is not possible to just go ahead and patch-reboot production systems like you do to systems in an enterprise environment. Production IT systems control your plant and can’t be taken offline in an uncontrolled/unplanned manner. Many run 24×7, so there is no after-hours window.
Solid planning and user involvement for applying updates and patches is therefore essential. This is not like the enterprise environment, where the user has little to no say at all. IT maintenance becomes a team sport, something many enterprise IT departments seem to be unfamiliar with, in my own experience.
Production IT and enterprise IT teams must work together
Old systems make it even more important that both teams work together. Systems or components may or may not be replaceable. Maybe production IT lacks the skills to do it while enterprise IT has those skills. Sometimes it’s the other way around.
The two IT teams must work together.
If systems must run despite them being a security risk, both teams must work together for a solution. Because you can’t simply go ahead and turn off or disconnect systems you “may not like”.
Steps to improve cyber security at all levels
Enterprise IT has to learn about production IT requirements, about the limitations and boundaries that exist around process automation.
Production IT, even if they are under stress to keep the plant running with minimal cost, must learn, understand and accept that they are part of a greater environment which must follow the same cyber security rules as all of the other systems in this environment.
A chain is only as strong as its weakest link.
To help both departments to achieve their goals, new products and regulations may need to be developed. Both IT departments and vendors have to closely collaborate to close existing gaps.
If enterprise IT and production IT come together, combine their knowledge and work together to develop a more secure, reliable, cost efficient, scalable and future-proof IT environment, they will pave the way to Smart Manufacturing.
I’ve seen only a couple of handfuls of companies, yet I have been shocked by the lack of good security. I’d love to hear about what you have seen.
What’s the worst cyber security risk that you have seen in a company, either in production IT or enterprise IT?