In Part 1 of this series, I walked through the background of the NERC CIP v5 controls and outlined what needs to be monitored for NERC CIP software requirements. In this final part of the series, we will take what we have learned and explore approaches for meeting the requirements, while considering security value. NERC CIP is supposed to be for security, after all!
At a high level, Tripwire’s Whitelist Profiler (WLP), a Tripwire Enterprise (TE) product extension, has many of the features needed for meeting the software monitoring requirements but process is important, as well. Additionally, in some cases, there are multiple approaches to a requirement, so the entity gets to choose what fits best for their interpretation and process.
OS and Firmware Version
OS version is generally tracked one of two ways, but both are easy. With a strict change management approach, TE can read the OS version and show if there are changes, which of course are very rare. A more scalable approach with policy is to also test the OS version, so the result flows into a unified view of compliance.
Probably the slickest approach is to monitor the OS version with WLP’s so-called “additional software” feature. The OS version will be reported right along with other software, or the implementation can even be broken out to show OS version per its CIP part number.
Firmware version is similarly monitored by reading it from the device. Add to that a test in TE and the control can be represented in a graphical summary red/green compliance view. The main variance customers ask to accommodate is during firmware updates across a fleet of devices where more than one version would be considered acceptable. This use case, however, is easily accommodated with a policy test.
Firmware monitoring gets particularly interesting when considering endpoints Tripwire cannot or perhaps should not connect to. Some substation endpoints cannot robustly support building and/or closing a telnet or SSH (secure shell) connection. In such a case, a trusted intermediary can be used to gather firmware version and other configuration details, as well as relay the data back to Tripwire for unified reporting. (The options here could easily become another blog entry.)
And finally, if a device is not reachable even by an intermediary tool, a person can paste the configuration into a designated location reachable by Tripwire. One customer even built a web frontend for this use case. Until all devices are somehow reachable in this case, the manual step unfortunately seems required but at least the device configuration is evaluated by standard controls and reported alongside all other devices.
Intentionally Installed Software
There is a two-pronged approach for this requirement. Most intentionally installed software registers itself (on Windows), or is installed by RPM. WLP will capture these as designed. PuTTY, OSI Monarch and Oracle on Linux are examples that would not be captured by WLP out of the box as they don’t register with the OS.
The so-called “additional software” feature simply would need to be configured to identify these applications. Internet Explorer can also be included in this scope although most entities consider it as part of the Windows OS.
Custom software: What was…
Prior to WECC’s announcement (referenced in my first blog post) about how it was going to interpret the requirement, the effort across regions was to implement some method of searching key portions of the file system for executable files. Portions of the file system that typically house intentionally installed software would often be excluded from the search.
While this smaller scope is more scalable to scan, it is not as secure. Still, the scanning usually uncovers anywhere from 50 to 200 files per system, which then inspires a healthy discussion about how to manage the files: put them in a standard location for instance, or consider eliminating some.
For one customer, the scanning for custom software had been in place for a few weeks on some production systems. When we looked at the scan history of one system, we could see an admin had done some PowerShell scripting on the desktop, placed some .ps files on the recycle bin, and not emptied the recycle bin. Additionally, a major software package had been deployed in a non-standard location and also happened to include a few copies of PuTTY. Good to know!
I encourage every entity to check with their region about how to interpret “custom” software. For non-NERC compliance purposes, I encourage customers to consider using the custom software monitoring control as it strikes a balance between scalable and informative.
The population of executable files on a system can be “large,” but it’s also generally pretty stable. Thus, a monitoring approach based on change management tends to scale well.
Custom software: What is…
At least in western audit region of WECC, the guidance has been to not worry about the “may include scripts” language in the Guidelines and Technical Basis section of the requirement document and leave it to the entity to determine for themselves what scripts or applications should be listed as “custom.”
In practice, the scope has amounted to a very short or zero-length list of in-house developed utilities. The cleanest approach I have seen is to use WLP software monitoring feature in such a way as to report just on these few, in-house developed utilities. This approach results in identical reporting as is produced for intentionally installed software.
A staple part of the approach is to implement a rule in TE to list out the installed patches. This simple step documents system state with a listing of the patches and any day-to-day changes that work well for change management objectives. Patches can be listed in one block of text and tracked as such, or they can be tracked as individual items (elements), as per the preference of the entity. Customers sometimes also use third-party tools for patch reporting (e.g. WSUS for Windows or Satellite for Linux).
Example Implementation of WLP Software Rules for Windows:
Here is a summary of the approaches with comments about how widespread the approaches are:
Prevalence of the Practice
|OS||WLP “additional software” feature, OR
A TE test of an OS-appropriate command that shows the version (e.g. “ver” on Windows).
|About 50/50 between the two options.|
|Firmware||Use a TE test of the output from a Command Output Validation Rule (COVR). Firmware version is usually listed it the begging of the output from a device-appropriate “show” command.
Optional: monitor BIOS version with WLP “additional software” feature.
|The approach using a test is predominant. I am not aware of any entities reporting BIOS versions.|
|Intentionally installed (registered, or through RPM)||WLP, out of the box feature.||This is the predominant approach.|
|Intentionally installed (not registered or RPM)||WLP “additional software” feature, out of the box, plus add your customizations to the additional_software.xml file.||This is the predominant approach.|
|Custom software||WLP “additional software” feature, add comment in the whitelist that the item is custom software, OR
For separate reporting by requirement part, create a dedicated instance of WLP software rule and CSV for customer software.
For extra credit, consider TE rules for script discovery, in combination with defining a process for elevating discovered software to “intentionally installed” and monitored by WLP. Can be used for CIP-010 R2 as well.
|In WECC at least, the trend is towards WLP. Some level of system scanning for executables was the predecessor approach.|
|Security patches||Monitor changes to all patches with base TE as part of the baseline (use a Command Output Capture Rule, COCR), AND often:
Rely on patch reporting from WSUS or similar tool, OR
Use WLP additional software option, listing all patches, OR
Wait for an enhancement to WLP so Windows patch reporting is 100% coverage.
|The trend is towards reliance on WLP instead of third party tools or a TE-only approach.|
To conclude, for NERC registered entities, the good news is that the problem of software monitoring has been solved. There are some interpretation and process issues to work out, but automation is at hand. You can get there from here!
For non-NERC readers, if you have a security concern around what is installed in your environment, consider adapting an approach used by the electric utility community in a process that fits for your organization.
Knowing what is actually in your environment is always the first step towards securing the environment.
To read part 1, click here.
 As observed on CIP assets in production.